Video display apparatus and operating method thereof

ABSTRACT

An operating method of a video display device is provided. The method according to one embodiment includes receiving a broadcasting signal through a tuner; obtaining an uncompressed audio or video (AV) content, through a receiving unit, other than the broadcasting signal; outputting the uncompressed AV content through a display unit; extracting characteristic information based on a genre of the uncompressed AV content; receiving a response to a query which includes the characteristic information; and outputting an enhanced service, including a service for recommending, based on the response.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of co-pending U.S. patent applicationSer. No. 14/362,832 filed on Jun. 4, 2014, which is the national phaseof PCT International Application No. PCT/KR2014/001354 filed on Feb. 19,2014, and which claims the benefit of U.S. Provisional Application No.61/767,256 filed on Feb. 21, 2013. The entire contents of all of theabove applications are hereby incorporated by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present disclosure relates to a video display device and a method ofoperating the same.

2. Discussion of the Related Art

As digital broadcasting is paving the way for its extensive spread, abroadcasting station transmits both main audio-visual (AV) contents andenhanced service data, which may be linked to the main audio and videocontents to provide information and services or may be separatelyprovided for other purposes.

However, a video display device in each home may unlikely receivebroadcast signals directly through air under a current broadcastingenvironment. Rather, most of cases, a video display device in each homeis connected to a broadcast receiving device to display uncompressedaudio-visual contents that the broadcast receiving device provides.

Additionally, the broadcast receiving device receives contents from aserver (called a Multichannel Video Programming Distributor (MVPD)). TheMVPD receives a broadcast signal from a broadcasting station, extractscontents from the received broadcast signal, converts the extractedcontent into signals having a proper format for transmission, andprovides the converted signals to a broadcast receiving device. Duringthese processes, the MVPD may exclude extracted enhanced service data ormay add another enhanced service data, so that the broadcastingreceiving device is compelled to receive an enhanced service dependenton the MVPD.

The MVPD provides a plurality of channels, which are provided bycontents providers, to a broadcast receiving device.

Since the broadcast receiving device extracts main AV data from signalreceived from the MVPD and provides only uncompressed audio-visual datato a video display device, only enhanced services provided from thebroadcast receiving device not the video display device are available.

SUMMARY OF THE INVENTION

Embodiments provide a video display device enabling correct contentrecognition and a method for operating the same.

In one embodiment, a operating method of a video display deviceincluding a display unit, the method comprising: obtaining anuncompressed audio and video content; extracting a first characteristicinformation corresponding to a first characteristic information typefrom the uncompressed audio and video content; receiving a firstresponse to a first query which includes the first characteristicinformation; determining a second characteristic information type basedon the first response; extracting a second characteristic informationcorresponding to the second characteristic information type from theuncompressed audio and video content; and receiving a second response toa second query which includes the second characteristic information.

The second characteristic information type may be determined based on agenre of the uncompressed audio and video content.

The first response to the first query may include a content identifierof the uncompressed audio and video and the genre of the uncompressedaudio and video content may be obtained based on the content identifier.

The determining the second characteristic information type may comprisestransmitting a third query which includes the content identifier anddetermining the second characteristic information type based on the athird response to the third query.

The third response to the third query may include the genre of theuncompressed audio and video content, and the determining the secondcharacteristic information type based on the a third response to thethird query may comprise determining the second characteristicinformation type based on the genre of the uncompressed audio and videocontent.

A video display device according to an embodiment may correctly identifycontent by changing a characteristic information type included in an ACRquery based on a genre of the content.

A video display device according to an embodiment may correctly identifycontent by changing a characteristic information type included in an ACRquery based on a content recognition accuracy.

A video display device according to an embodiment may correctly identifycontent by changing a characteristic information type included in an ACRquery based on a constraint on the video display device.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating the network topology according tothe embodiment.

FIG. 2 is a block diagram illustrating a watermark based networktopology according to an embodiment.

FIG. 3 is a ladder diagram illustrating a data flow in a watermark basednetwork topology according to an embodiment.

FIG. 4 is view illustrating a watermark based content recognition timingaccording to an embodiment.

FIG. 5 is a block diagram illustrating a fingerprint based networktopology according to an embodiment.

FIG. 6 is a ladder diagram illustrating a data flow in a fingerprintbased network topology according to an embodiment.

FIG. 7 is a view illustrating an XML schema diagram of ACR-Result typecontaining a query result according to an embodiment.

FIG. 8 is a block diagram illustrating a watermark and fingerprint basednetwork topology according to an embodiment.

FIG. 9 is a ladder diagram illustrating a data flow in a watermark andfingerprint based network topology according to an embodiment.

FIG. 10 is a block diagram illustrating the video display deviceaccording to the embodiment.

FIG. 11 is a flowchart illustrating a method of synchronizing a playbacktime of a main audio and video content with a playback time of anenhanced service according to an embodiment.

FIG. 12 is a conceptual diagram illustrating a method of synchronizing aplayback time of a main audio and video content with a playback time ofan enhanced service according to an embodiment.

FIG. 13 is a screen when an auto playing application is executedaccording to an embodiment.

FIG. 14 is a screen when a content provider application is executedaccording to an embodiment.

FIG. 15 is a view of a user interface for controlling an ACR functionaccording to an embodiment.

FIG. 16 is a flowchart illustrating a method of controlling a videodisplay device according to the embodiment.

FIG. 17 is a view illustrating a network topology according to anembodiment of the present invention.

FIG. 18 is a view illustrating a Live+ app according to variousembodiments of the present invention.

FIG. 19 is a view illustrating a method of executing a Live+ appaccording to various embodiments of the present invention.

FIG. 20 is a view illustrating a tab function of a Live+ app accordingto an embodiment of the present invention.

FIG. 21 is a view illustrating an architecture of the third party server90 according to an embodiment of the present invention.

FIG. 22 is a ladder diagram illustrating an operation of a video displaydevice for an automatic execution application according to an embodimentof the present invention.

FIG. 23 is a ladder diagram illustrating an operation of a video displaydevice for a channel-independent application according to an embodimentof the present invention.

FIG. 24 is a ladder diagram illustrating an operation of a video displaydevice for a channel-independent application according to anotherembodiment of the present invention.

FIG. 25 is a conceptual diagram illustrating a video display deviceaccording to an embodiment of the present invention.

FIG. 26 is a ladder diagram illustrating an operation of a video displaydevice according to another embodiment of the present invention.

FIG. 27 is an XML schema diagram of ACR configuration informationaccording to an embodiment of the present invention.

FIG. 28 is a ladder diagram illustrating an operation of a video displaydevice according to another embodiment of the present invention.

FIG. 29 is an XML schema diagram of ACR configuration informationaccording to an embodiment of the present invention.

FIG. 30 is a view of an EPG window displaying the MVPD logo of DishNetwork, a satellite operator, according to an embodiment of the presentinvention.

FIG. 31 is a view of an EPG window displaying the MVPD logo of Comcast,a Cable operator, according to an embodiment of the present invention.

FIG. 32 is a view of an EPG window displaying the MVPD logo of Comcast,a Cable operator, according to an embodiment of the present invention.

FIG. 33 is a view of a menu window displaying the MVPD logo of DirectTV,a satellite operator, according to an embodiment of the presentinvention.

FIG. 34 is a view of channel map processing according to an embodimentof the present invention.

FIG. 35 is a view of channel map processing according to an embodimentof the present invention.

FIG. 36 is a conceptual diagram of a video display device according toan embodiment of the present invention.

FIG. 37 is a conceptual diagram of a video display device according toan embodiment of the present invention.

FIG. 38 is a block diagram illustrating a structure of a fingerprintbased video display device according to another embodiment.

FIG. 39 is a block diagram illustrating a structure of a watermark basedvideo display device according to another embodiment.

FIG. 40 is an XML schema diagram of ACR-Resulttype containing a queryresult according to another embodiment.

FIG. 41 is a diagram illustrating a network topology in which a videodisplay device determines a characteristic information type according toan embodiment.

FIG. 42 is a ladder diagram illustrating that a video display devicedetermines a characteristic information type according to an embodiment.

FIG. 43 is a flowchart illustrating that a video display device changesthe characteristic information type based on the accuracy of contentrecognition according to an embodiment.

FIG. 44 is a diagram illustrating a network topology in which a videodisplay device determines a characteristic information type according toanother embodiment.

FIG. 45 is a ladder diagram illustrating that a video display devicedetermines a characteristic information type according to anotherembodiment.

FIG. 46 is a conceptual diagram illustrating a video display deviceaccording to an embodiment.

FIG. 47 is a block diagram illustrating a structure of a fingerprintbased video display device according to another embodiment.

FIG. 48 is a block diagram illustrating a structure of a fingerprintbased video display device according to another embodiment.

DETAILED DESCRIPTION OF THE INVENTION

Hereinafter, a mobile terminal relating to the present invention will bedescribed in more detail with reference to the accompanying drawings.Noun suffixes such as “engine”, “module”, and “unit” for components indescription below are given or mixed in consideration of easiness inwriting the specification. That is, the noun suffixes themselves doesnot have respectively distinguishable meanings or roles.

A network topology will be described with reference to FIGS. 1 to 9according to an embodiment.

FIG. 1 is a block diagram illustrating the network topology according tothe embodiment.

As shown in FIG. 1, the network topology includes a content providingserver 10, a content recognizing service providing server 20, a multichannel video distributing server 30, an enhanced service informationproviding server 40, a plurality of enhanced service providing servers50, a broadcast receiving device 60, a network 70, and a video displaydevice 100.

The content providing server 10 may correspond to a broadcasting stationand broadcasts a broadcast signal including main audio-visual contents.The broadcast signal may further include enhanced services. The enhancedservices may or may not relate to main audio-visual contents. Theenhanced services may have formats such as service information,metadata, additional data, compiled execution files, web applications,Hypertext Markup Language (HTML) documents, XML documents, CascadingStyle Sheet (CSS) documents, audio files, video files, ATSC 2.0contents, and addresses such as Uniform Resource Locator (URL). Theremay be at least one content providing server.

The content recognizing service providing server 20 provides a contentrecognizing service that allows the video display device 100 torecognize content on the basis of main audio-visual content. The contentrecognizing service providing server 20 may or may not edit the mainaudio-visual content. There may be at least one content recognizingservice providing server.

The content recognizing service providing server 20 may be a watermarkserver that edits the main audio-visual content to insert a visiblewatermark, which may look a logo, into the main audio-visual content.This watermark server may insert the logo of a content provider at theupper-left or upper-right of each frame in the main audio-visual contentas a watermark.

Additionally, the content recognizing service providing server 20 may bea watermark server that edits the main audio-visual content to insertcontent information into the main audio-visual content as an invisiblewatermark.

Additionally, the content recognizing service providing server 20 may bea fingerprint server that extracts characteristic information from someframes or audio samples of the main audio-visual content and stores it.This characteristic information is called signature.

The multi channel video distributing server 30 receives and multiplexesbroadcast signals from a plurality of broadcasting stations and providesthe multiplexed broadcast signals to the broadcast receiving device 60.Especially, the multi channel video distributing server 30 performsdemodulation and channel decoding on the received broadcast signals toextract main audio-visual content and enhanced service, and then,performs channel encoding on the extracted main audio-visual content andenhanced service to generate a multiplexed signal for distribution. Atthis point, since the multi channel video distributing server 30 mayexclude the extracted enhanced service or may add another enhancedservice, a broadcasting station may not provide services led by it.There may be at least one multi channel video distributing server.

The broadcasting device 60 may tune a channel selected by a user andreceives a signal of the tuned channel, and then, performs demodulationand channel decoding on the received signal to extract a mainaudio-visual content. The broadcasting device 60 decodes the extractedmain audio-visual content through H.264/Moving Picture Experts Group-4advanced video coding (MPEG-4 AVC), Dolby AC-3 or Moving Picture ExpertsGroup-2 Advanced Audio Coding (MPEG-2 AAC) algorithm to generate anuncompressed main audio-visual (AV) content. The broadcast receivingdevice 60 provides the generated uncompressed main audio and videocontent to the video display device 100 through its external input port.

The enhanced service information providing server 40 provides enhancedservice information on at least one available enhanced service relatingto a main audio and video content in response to a request of a videodisplay device. There may be at least one enhanced service providingserver. The enhanced service information providing server 40 may provideenhanced service information on the enhanced service having the highestpriority among a plurality of available enhanced services.

The enhanced service providing server 50 provides at least one availableenhanced service relating to a main audio and video content in responseto a request of a video display device. There may be at least oneenhanced service providing server.

The video display device 100 may be a television, a notebook computer, ahand phone, and a smart phone, each including a display unit. The videodisplay device 100 may receive an uncompressed main audio and videocontent from the broadcast receiving device 60 or a broadcast signalincluding an encoded main audio and video content from the contentsproviding server 10 or the multi channel video distributing server 30.The video display device 100 may receive a content recognizing servicefrom the content recognizing service providing server 20 through thenetwork 70, an address of at least one available enhanced servicerelating to a main audio and video content from the enhanced serviceinformation providing server 40 through the network 70, and at least oneavailable enhanced service relating to a main audio and video contentfrom the enhanced service providing server 50.

At least two of the content providing server 10, the content recognizingservice providing server 20, the multi channel video distributing server30, the enhanced service information providing server 40, and theplurality of enhanced service providing servers 50 may be combined in aform of one server and may be operated by one provider.

FIG. 2 is a block diagram illustrating a watermark based networktopology according to an embodiment.

As shown in FIG. 2, the watermark based network topology may furtherinclude a watermark server 21.

As shown in FIG. 2, the watermark server 21 edits a main audio and videocontent to insert content information into it. The multi channel videodistributing server 30 may receive and distribute a broadcast signalincluding the modified main audio and video content. Especially, awatermark server may use a digital watermarking technique describedbelow.

A digital watermark is a process for inserting information, which may bealmost undeletable, into a digital signal. For example, the digitalsignal may be audio, picture, or video. If the digital signal is copied,the inserted information is included in the copy. One digital signal maycarry several different watermarks simultaneously.

In visible watermarking, the inserted information may be identifiable ina picture or video. Typically, the inserted information may be a text orlogo identifying a media owner. If a television broadcasting stationadds its logo in a corner of a video, this is an identifiable watermark.

In invisible watermarking, although information as digital data is addedto audio, picture, or video, a user may be aware of a predeterminedamount of information but may not recognize it. A secret message may bedelivered through the invisible watermarking.

One application of the watermarking is a copyright protection system forpreventing the illegal copy of digital media. For example, a copy deviceobtains a watermark from digital media before copying the digital mediaand determines whether to copy or not on the bases of the content of thewatermark.

Another application of the watermarking is source tracking of digitalmedia. A watermark is embedded in the digital media at each point of adistribution path. If such digital media is found later, a watermark maybe extracted from the digital media and a distribution source may berecognized from the content of the watermark.

Another application of invisible watermarking is a description fordigital media.

A file format for digital media may include additional informationcalled metadata and a digital watermark is distinguished from metadatain that it is delivered as an AV signal itself of digital media.

The watermarking method may include spread spectrum, quantization, andamplitude modulation.

If a marked signal is obtained through additional editing, thewatermarking method corresponds to the spread spectrum. Although it isknown that the spread spectrum watermark is quite strong, not muchinformation is contained because the watermark interferes with anembedded host signal.

If a marked signal is obtained through the quantization, thewatermarking method corresponds to a quantization type. The quantizationwatermark is weak, much information may be contained.

If a marked signal is obtained through an additional editing methodsimilar to the spread spectrum in a spatial domain, a watermarkingmethod corresponds to the amplitude modulation.

FIG. 3 is a ladder diagram illustrating a data flow in a watermark basednetwork topology according to an embodiment.

First, the content providing server 10 transmits a broadcast signalincluding a main audio and video content and an enhanced service inoperation S101.

The watermark server 21 receives a broadcast signal that the contentproviding server 10 provides, inserts a visible watermark such as a logoor watermark information as an invisible watermark into the main audioand video content by editing the main audio and video content, andprovides the watermarked main audio and video content and enhancedservice to the MVPD 30 in operation S103.

The watermark information inserted through an invisible watermark mayinclude at least one of a watermark purpose, content information,enhanced service information, and an available enhanced service. Thewatermark purpose represents one of illegal copy prevention, viewerratings, and enhanced service acquisition.

The content information may include at least one of identificationinformation of a content provider that provides main audio and videocontent, main audio and video content identification information, timeinformation of a content section used in content informationacquisition, names of channels through which main audio and videocontent is broadcasted, logos of channels through which main audio andvideo content is broadcasted, descriptions of channels through whichmain audio and video content is broadcasted, a usage informationreporting period, the minimum usage time for usage informationacquisition, and available enhanced service information relating to mainaudio and video content.

If the video display device 100 uses a watermark to acquire contentinformation, the time information of a content section used for contentinformation acquisition may be the time information of a content sectioninto which a watermark used is embedded. If the video display device 100uses a fingerprint to acquire content information, the time informationof a content section used for content information acquisition may be thetime information of a content section where characteristic informationis extracted. The time information of a content section used for contentinformation acquisition may include at least one of the start time of acontent section used for content information acquisition, the durationof a content section used for content information acquisition, and theend time of a content section used for content information acquisition.

The usage information reporting address may include at least one of amain audio and video content watching information reporting address andan enhanced service usage information reporting address. The usageinformation reporting period may include at least one of a main audioand video content watching information reporting period and an enhancedservice usage information reporting period. A minimum usage time forusage information acquisition may include at least one of a minimumwatching time for a main audio and video content watching informationacquisition and a minimum usage time for enhanced service usageinformation extraction.

On the basis that a main audio and video content is watched for morethan the minimum watching time, the video display device 100 acquireswatching information of the main audio and video content and reports theacquired watching information to the main audio and video contentwatching information reporting address in the main audio and videocontent watching information reporting period.

On the basis that an enhanced service is used for more than the minimumusage time, the video display device 100 acquires enhanced service usageinformation and reports the acquired usage information to the enhancedservice usage information reporting address in the enhanced serviceusage information reporting period.

The enhanced service information may include at least one of informationon whether an enhanced service exists, an enhanced service addressproviding server address, an acquisition path of each available enhancedservice, an address for each available enhanced service, a start time ofeach available enhanced service, an end time of each available enhancedservice, a lifetime of each available enhanced service, an acquisitionmode of each available enhanced service, a request period of eachavailable enhanced service, priority information each available enhancedservice, description of each available enhanced service, a category ofeach available enhanced service, a usage information reporting address,a usage information reporting period, and the minimum usage time forusage information acquisition.

The acquisition path of available enhanced service may be representedwith IP or Advanced Television Systems Committee-Mobile/Handheld (ATSCM/H). If the acquisition path of available enhanced service is ATSC M/H,enhanced service information may further include frequency informationand channel information. An acquisition mode of each available enhancedservice may represent Push or Pull.

Moreover, the watermark server 21 may insert watermark information as aninvisible watermark into the logo of a main audio and video content.

For example, the watermark server 21 may insert a barcode at apredetermined position of a logo. At this point, the predeterminedposition of the logo may correspond to the first line at the bottom ofan area where the logo is displayed. The video display device 100 maynot display a barcode when receiving a main audio and video contentincluding a logo with the barcode inserted.

For example, the watermark server 21 may insert a barcode at apredetermined position of a logo. At this point, the log may maintainits form.

For example, the watermark server 21 may insert N-bit watermarkinformation at each of the logos of M frames. That is, the watermarkserver 21 may insert M*N watermark information in M frames.

The MVPD 30 receives broadcast signals including watermarked main audioand video content and enhanced service and generates a multiplexedsignal to provide it to the broadcast receiving device 60 in operationS105. At this point, the multiplexed signal may exclude the receivedenhanced service or may include new enhanced service.

The broadcast receiving device 60 tunes a channel that a user selectsand receives signals of the tuned channel, demodulates the receivedsignals, performs channel decoding and AV decoding on the demodulatedsignals to generate an uncompressed main audio and video content, andthen, provides the generated uncompressed main audio and video contentto the video display device 100 in operation S106.

Moreover, the content providing server 10 also broadcasts a broadcastsignal including a main audio and video content through a wirelesschannel in operation S107.

Additionally, the MVPD 30 may directly transmit a broadcast signalincluding a main audio and video content to the video display device 100without going through the broadcast receiving device 60 in operationS108.

The video display device 100 may receive an uncompressed main audio andvideo content through the broadcast receiving device 60. Additionally,the video display device 100 may receive a broadcast signal through awireless channel, and then, may demodulate and decode the receivedbroadcast signal to obtain a main audio and video content. Additionally,the video display device 100 may receive a broadcast signal from theMVPD 30, and then, may demodulate and decode the received broadcastsignal to obtain a main audio and video content. The video displaydevice 100 extracts watermark information from some frames or a sectionof audio samples of the obtained main audio and video content. Ifwatermark information corresponds to a logo, the video display device100 confirms a watermark server address corresponding to a logoextracted from a corresponding relationship between a plurality of logosand a plurality of watermark server addresses. When the watermarkinformation corresponds to the logo, the video display device 100 cannotidentify the main audio and video content only with the logo.Additionally, when the watermark information does not include contentinformation, the video display device 100 cannot identify the main audioand video content but the watermark information may include contentprovider identifying information or a watermark server address. When thewatermark information includes the content provider identifyinginformation, the video display device 100 may confirm a watermark serveraddress corresponding to the content provider identifying informationextracted from a corresponding relationship between a plurality ofcontent provider identifying information and a plurality of watermarkserver addresses. In this manner, when the video display device 100cannot identify a main audio and video content the video display device100 only with the watermark information, it accesses the watermarkserver 21 corresponding to the obtained watermark server address totransmit a first query in operation S109.

The watermark server 21 provides a first reply to the first query inoperation S111. The first reply may include at least one of contentinformation, enhanced service information, and an available enhancedservice.

If the watermark information and the first reply do not include anenhanced service address, the video display device 100 cannot obtainenhanced service. However, the watermark information and the first replymay include an enhanced service address providing server address. Inthis manner, the video display device 100 does not obtain a serviceaddress or enhanced service through the watermark information and thefirst reply. If the video display device 100 obtains an enhanced serviceaddress providing server address, it accesses the enhanced serviceinformation providing server 40 corresponding to the obtained enhancedservice address providing server address to transmit a second queryincluding content information in operation S119.

The enhanced service information providing server 40 searches at leastone available enhanced service relating to the content information ofthe second query. Later, the enhanced service information providingserver 40 provides to the video display device 100 enhanced serviceinformation for at least one available enhanced service as a secondreply to the second query in operation S121.

If the video display device 100 obtains at least one available enhancedservice address through the watermark information, the first reply, orthe second reply, it accesses the at least one available enhancedservice address to request enhanced service in operation S123, and then,obtains the enhanced service in operation S125.

FIG. 4 is a view illustrating a watermark based content recognitiontiming according to an embodiment.

As shown in FIG. 4, when the broadcast receiving device 60 is turned onand tunes a channel, and also, the video display device 100 receives amain audio and video content of the turned channel from the broadcastreceiving device 60 through an external input port 111, the videodisplay device 100 may sense a content provider identifier (or abroadcasting station identifier) from the watermark of the main audioand video content. Then, the video display device 100 may sense contentinformation from the watermark of the main audio and video content onthe basis of the sensed content provider identifier.

At this point, as shown in FIG. 4, the detection available period of thecontent provider identifier may be different from that of the contentinformation. Especially, the detection available period of the contentprovider identifier may be shorter than that of the content information.Through this, the video display device 100 may have an efficientconfiguration for detecting only necessary information.

FIG. 5 is a block diagram illustrating a fingerprint based networktopology according to an embodiment.

As shown in FIG. 5, the network topology may further include afingerprint server 22.

As shown in FIG. 5, the fingerprint server 22 does not edit a main audioand video content, but extracts characteristic information from someframes or a section of audio samples of the main audio and video contentand stores the extracted characteristic information. Then, whenreceiving the characteristic information from the video display device100, the fingerprint server 22 provides an identifier and timeinformation of an audio and video content corresponding to the receivedcharacteristic information.

FIG. 6 is a ladder diagram illustrating a data flow in a fingerprintbased network topology according to an embodiment.

First, the content providing server 10 transmits a broadcast signalincluding a main audio and video content and an enhanced service inoperation S201.

The fingerprint server 22 receives a broadcast signal that the contentproviding server 10, extracts a plurality of pieces of characteristicinformation from a plurality of frame sections or a plurality of audiosections of the main audio and video content, and establishes a databasefor a plurality of query results corresponding to the plurality ofcharacteristic information in operation S203. The query result mayinclude at least one of content information, enhanced serviceinformation, and an available enhanced service.

The MVPD 30 receives broadcast signals including a main audio and videocontent and enhanced service and generates a multiplexed signal toprovide it to the broadcast receiving device 60 in operation S205. Atthis point, the multiplexed signal may exclude the received enhancedservice or may include new enhanced service.

The broadcast receiving device 60 tunes a channel that a user selectsand receives signals of the tuned channel, demodulates the receivedsignals, performs channel decoding and AV decoding on the demodulatedsignals to generate an uncompressed main audio and video content, andthen, provides the generated uncompressed main audio and video contentto the video display device 100 in operation S206.

Moreover, the content providing server 10 also broadcasts a broadcastsignal including a main audio and video content through a wirelesschannel in operation S207.

Additionally, the MVPD 30 may directly transmit a broadcast signalincluding a main audio and video content to the video display device 100without going through the broadcast receiving device 60.

The video display device 100 may receive an uncompressed main audio andvideo content through the broadcast receiving device 60. Additionally,the video display device 100 may receive a broadcast signal through awireless channel, and then, may demodulate and decode the receivedbroadcast signal to obtain a main audio and video content. Additionally,the video display device 100 may receive a broadcast signal from theMVPD 30, and then, may demodulate and decode the received broadcastsignal to obtain a main audio and video content. The video displaydevice 100 extracts characteristic information from some frames or asection of audio samples of the obtained main audio and video content inoperation S213.

The video display device 100 accesses the fingerprint server 22corresponding to the predetermined fingerprint server address totransmit a first query including the extracted characteristicinformation in operation S215.

The fingerprint server 22 provides a query result as a first reply tothe first query in operation S217. If the first reply corresponds tofail, the video display device 100 accesses the fingerprint server 22corresponding to another fingerprint server address to transmit a firstquery including the extracted characteristic information.

The fingerprint server 22 may provide Extensible Markup Language (XML)document as a query result. Examples of the XML document containing aquery result will be described with reference to FIG. 7 and Table 1.

FIG. 7 is a view illustrating an XML schema diagram of ACR-Resulttypecontaining a query result according to an embodiment.

As shown in FIG. 7, ACR-Resulttype containing a query result includesResultCode attributes and ContentID, NTPTimestamp,SignalingChannelInformation, and ServiceInformation elements.

For example, if the ResultCode attribute has 200, this may mean that thequery result is successful. For example, if the ResultCode attribute has404, this may mean that the query result is unsuccessful.

The SignalingChannelInformation element includes a SignalingChannelURL,and the SignalingChannelURL element includes an UpdateMode andPollingCycle attributes. The UpdateMode attribute may have a Pull valueor a Push value.

The ServiceInformation element includes ServiceName, ServiceLogo, andServiceDescription elements.

Table 1 illustrates an XML schema of ACR-ResultType containing the queryresult.

TABLE 1 <xs:complexType name=“ACR-ResultType”>   <xs:sequence>   <xs:element name=“ContentID” type=“xs:anyURI”/>    <xs:elementname=“NTPTimestamp” type=“xs:unsignedLong”/>    <xs:elementname=“SignalingChannelInformation”>     <xs:complexType>      <xs:sequence>        <xs:element name=“SignalingChannelURL”maxOccurs=“unbounded”>         <xs:complexType>         <xs:simpleContent>           <xs:extension base=“xs:anyURI”>           <xs:attribute name=“UpdateMode”>             <xs:simpleType>             <xs:restriction base=“xs:string”>              <xs:enumeration value=“Pull”/>              <xs:enumeration value=“Push”/>             </xs:restriction>             </xs:simpleType>           </xs:attribute>            <xs:attribute name=“PollingCycle”type=“xs:unsignedInt”/>           </xs:extension>         </xs:simpleContent>         </xs:complexType>       </xs:element>       </xs:sequence>      </xs:complexType>    </xs:element>     <xs:element name=“ServiceInformation”>     <xs:complexType>       <xs:sequence>        <xs:elementname=“ServiceName” type=“xs:string”/>        <xs:elementname=“ServiceLogo” type=“xs:anyURI” minOccurs=“0”/>        <xs:elementname=“ServiceDescription” type=“xs:string” minOccurs=“0”maxOccurs=“unbounded”/>      </xs:sequence>     </xs:complexType>   </xs:element>    <xs:any namespace=“##other” processContents=“skip”minOccurs=“0” maxOccurs=“unbounded”/>   </xs:sequence>   <xs:attributename=“ResultCode” type=“xs:string” use=“required”/>   <xs:anyAttributeprocessContents=“skip”/>  </xs:complexType>

As the ContentID element, an ATSC content identifier may be used asshown in Table 2.

TABLE 2 Syntax The Number of bits format ATSC_content_identifier( ) {TSID 16 uimsbf reserved 2 bslbf end_of_day 5 uimsbf unique_for 9 uimsbfcontent_id var }

As shown in Table 2, the ATSC content identifier has a structureincluding TSID and a house number.

The 16 bit unsigned integer TSID carries a transport stream identifier.

The 5 bit unsigned integer end_of_day is set with an hour in a day ofwhen a content_id value can be reused after broadcasting is finished.

The 9 bit unsigned integer unique_for is set with the number of day ofwhen the content_id value cannot be reused.

Content_id represents a content identifier. The video display device 100reduces unique_for by 1 in a corresponding time to end_of_day daily andpresumes that content_id is unique if unique_for is not 0.

Moreover, as the ContentID element, a global service identifier forATSC-M/H service may be used as described below.

The global service identifier has the following form.

˜urn:oma:bcast:iauth:atsc:service:<region>:<xsid>:<serviceid>

Here, <region> is an international country code including two charactersregulated by ISO 639-2. <xsid> for local service is a decimal number ofTSID as defined in <region>, and <xsid> (regional service) (major >69)is “0”. <serviceid> is defined with <major> or <minor>. <major>represent a Major Channel number, and <minor> represents a Minor ChannelNumber.

Examples of the global service identifier are as follows.

-   -   urn:oma:bcast:iauth:atsc:service:us:1234:5.1    -   urn:oma:bcast:iauth:atsc:service:us:0:100.200

Moreover, as the ContentID element, an ATSC content identifier may beused as described below.

The ATSC content identifier has the following form.

urn:oma:bcast:iauth:atsc:content:<region>:<xsidz>:<contentid>:<unique_for>:<end_of_day>

Here, <region> is an international country code including two charactersregulated by ISO 639-2. <xsid> for local service is a decimal number ofTSID as defined in <region>, and may be followed by “.” <serviceid>.<xsid> for (regional service) (major >69) is <serviceid>. <content_id>is a base64 sign of a content_id field defined in Table 2, <unique_for>is a decimal number sign of an unique_for field defined in Table 2, and<end_of_day> is a decimal number sign of an end_of_day field defined inTable 2.

Hereinafter, FIG. 6 is described again.

If the query result does not include an enhanced service address orenhanced service but includes an enhanced service address providingserver address, the video display device 100 accesses the enhancedservice information providing server 40 corresponding to the obtainedenhanced service address providing server address to transmit a secondquery including content information in operation S219.

The enhanced service information providing server 40 searches at leastone available enhanced service relating to the content information ofthe second query. Later, the enhanced service information providingserver 40 provides to the video display device 100 enhanced serviceinformation for at least one available enhanced service as a secondreply to the second query in operation S221.

If the video display device 100 obtains at least one available enhancedservice address through the first reply or the second reply, it accessesthe at least one available enhanced service address to request enhancedservice in operation S223, and then, obtains the enhanced service inoperation S225.

When the UpdateMode attribute has a Pull value, the video display device100 transmits an HTTP request to the enhanced service providing server50 through SignalingChannelURL and receives an HTTP reply including aPSIP binary stream from the enhanced service providing server 50 inresponse to the request. In this case, the video display device 100 maytransmit the HTTP request according to a Polling period designated asthe PollingCycle attribute. Additionally, the SignalingChannelURLelement may have an update time attribute. In this case, the videodisplay device 100 may transmit the HTTP request according to an updatetime designated as the update time attribute.

If the UpdateMode attribute has a Push value, the video display device100 may receive update from a server asynchronously throughXMLHTTPRequest API. After the video display device 100 transmits anasynchronous request to a server through XMLHTTPRequest object, if thereis a change of signaling information, the server provides the signalinginformation as a reply through the channel. If there is limitation insession standby time, a server generates a session timeout reply and areceiver recognizes the generated timeout reply to transmit a requestagain, so that a signaling channel between the receiver and the servermay be maintained for all time.

FIG. 8 is a block diagram illustrating a watermark and fingerprint basednetwork topology according to an embodiment.

As shown in FIG. 8, the watermark and fingerprint based network topologymay further include a watermark server 21 and a fingerprint server 22.

As shown in FIG. 8, the watermark server 21 inserts content provideridentifying information into a main audio and video content. Thewatermark server 21 may insert content provider identifying informationas a visible watermark such as a logo or an invisible watermark into amain audio and video content.

The fingerprint server 22 does not edit a main audio and video content,but extracts characteristic information from some frames or a certainsection of audio samples of the main audio and video content and storesthe extracted characteristic information. Then, when receiving thecharacteristic information from the video display device 100, thefingerprint server 22 provides an identifier and time information of anaudio and video content corresponding to the received characteristicinformation.

FIG. 9 is a ladder diagram illustrating a data flow in a watermark andfingerprint based network topology according to an embodiment.

First, the content providing server 10 transmits a broadcast signalincluding a main audio and video content and an enhanced service inoperation S301.

The watermark server 21 receives a broadcast signal that the contentproviding server 10 provides, inserts a visible watermark such as a logoor watermark information as an invisible watermark into the main audioand video content by editing the main audio and video content, andprovides the watermarked main audio and video content and enhancedservice to the MVPD 30 in operation S303. The watermark informationinserted through an invisible watermark may include at least one ofcontent information, enhanced service information, and an availableenhanced service. The content information and enhanced serviceinformation are described above.

The MVPD 30 receives broadcast signals including watermarked main audioand video content and enhanced service and generates a multiplexedsignal to provide it to the broadcast receiving device 60 in operationS305. At this point, the multiplexed signal may exclude the receivedenhanced service or may include new enhanced service.

The broadcast receiving device 60 tunes a channel that a user selectsand receives signals of the tuned channel, demodulates the receivedsignals, performs channel decoding and AV decoding on the demodulatedsignals to generate an uncompressed main audio and video content, andthen, provides the generated uncompressed main audio and video contentto the video display device 100 in operation S306.

Moreover, the content providing server 10 also broadcasts a broadcastsignal including a main audio and video content through a wirelesschannel in operation S307.

Additionally, the MVPD 30 may directly transmit a broadcast signalincluding a main audio and video content to the video display device 100without going through the broadcast receiving device 60 in operationS308.

The video display device 100 may receive an uncompressed main audio andvideo content through the broadcast receiving device 60. Additionally,the video display device 100 may receive a broadcast signal through awireless channel, and then, may demodulate and decode the receivedbroadcast signal to obtain a main audio and video content. Additionally,the video display device 100 may receive a broadcast signal from theMVPD 30, and then, may demodulate and decode the received broadcastsignal to obtain a main audio and video content. The video displaydevice 100 extracts watermark information from audio samples in someframes or periods of the obtained main audio and video content. Ifwatermark information corresponds to a logo, the video display device100 confirms a watermark server address corresponding to a logoextracted from a corresponding relationship between a plurality of logosand a plurality of watermark server addresses. When the watermarkinformation corresponds to the logo, the video display device 100 cannotidentify the main audio and video content only with the logo.Additionally, when the watermark information does not include contentinformation, the video display device 100 cannot identify the main audioand video content but the watermark information may include contentprovider identifying information or a watermark server address. When thewatermark information includes the content provider identifyinginformation, the video display device 100 may confirm a watermark serveraddress corresponding to the content provider identifying informationextracted from a corresponding relationship between a plurality ofcontent provider identifying information and a plurality of watermarkserver addresses. In this manner, when the video display device 100cannot identify a main audio and video content the video display device100 only with the watermark information, it accesses the watermarkserver 21 corresponding to the obtained watermark server address totransmit a first query in operation S309.

The watermark server 21 provides a first reply to the first query inoperation S311. The first reply may include at least one of afingerprint server address, content information, enhanced serviceinformation, and an available enhanced service. The content informationand enhanced service information are described above.

If the watermark information and the first reply include a fingerprintserver address, the video display device 100 extracts characteristicinformation from some frames or a certain section of audio samples ofthe main audio and video content in operation S313.

The video display device 100 accesses the fingerprint server 22corresponding to the fingerprint server address in the first reply totransmit a second query including the extracted characteristicinformation in operation S315.

The fingerprint server 22 provides a query result as a second reply tothe second query in operation S317.

If the query result does not include an enhanced service address orenhanced service but includes an enhanced service address providingserver address, the video display device 100 accesses the enhancedservice information providing server 40 corresponding to the obtainedenhanced service address providing server address to transmit a thirdquery including content information in operation S319.

The enhanced service information providing server 40 searches at leastone available enhanced service relating to the content information ofthe third query. Later, the enhanced service information providingserver 40 provides to the video display device 100 enhanced serviceinformation for at least one available enhanced service as a third replyto the third query in operation S321.

If the video display device 100 obtains at least one available enhancedservice address through the first reply, the second reply, or the thirdreply, it accesses the at least one available enhanced service addressto request enhanced service in operation S323, and then, obtains theenhanced service in operation S325.

Then, referring to FIG. 10, the video display device 100 will bedescribed according to an embodiment.

FIG. 10 is a block diagram illustrating the video display deviceaccording to the embodiment.

As shown in FIG. 10, the video display device 100 includes a broadcastsignal receiving unit 101, a demodulation unit 103, a channel decodingunit 105, a demultiplexing unit 107, an AV decoding unit 109, anexternal input port 111, a play controlling unit 113, a play device 120,an enhanced service management unit 130, a data transmitting/receivingunit 141, and a memory 150.

The broadcast signal receiving unit 101 receives a broadcast signal fromthe content providing server 10 or MVPD 30.

The demodulation unit 103 demodulates the received broadcast signal togenerate a demodulated signal.

The channel decoding unit 105 performs channel decoding on thedemodulated signal to generate channel-decoded data.

The demultiplexing unit 107 separates a main audio and video content andenhanced service from the channel-decoded data. The separated enhancedservice is stored in an enhanced service storage unit 152.

The AV decoding unit 109 performs AV decoding on the separated mainaudio and video content to generate an uncompressed main audio and videocontent.

Moreover, the external input port 111 receives an uncompressed mainaudio and video content from the broadcast receiving device 60, adigital versatile disk (DVD) player, a Blu-ray disk player, and so on.The external input port 111 may include at least one of a DSUB port, aHigh Definition Multimedia interface (HDMI) port, a Digital VisualInterface (DVI) port, a composite port, a component port, and an S-Videoport.

The play controlling unit 113 controls the play device 120 to play atleast one of an uncompressed main audio and video content that the AVdecoding unit 109 generates and an uncompressed main audio and videocontent received from the external input port 111 according to a user'sselection.

The play device 120 includes a display unit 121 and a speaker 123. Thedisplay unit 21 may include at least one of a liquid crystal display(LCD), a thin film transistor-liquid crystal display (TFT LCD), anorganic light-emitting diode (OLED), a flexible display, and a 3Ddisplay.

The enhanced service management unit 130 obtains content information ofthe main audio and video content and obtains available enhanced serviceon the basis of the obtained content information. Especially, asdescribed above, the enhanced service management unit 130 may obtain theidentification information of the main audio and video content on thebasis of some frames or a certain section of audio samples theuncompressed main audio and video content. This is called automaticcontents recognition (ACR) in this specification.

The data transmitting/receiving unit 141 may include an AdvancedTelevision Systems Committee-Mobile/Handheld (ATSC-M/H) channeltransmitting/receiving unit 141 a and an IP transmitting/receiving unit141 b.

The memory 150 may include at least one type of storage medium such as aflash memory type, a hard disk type, a multimedia card micro type, acard type memory such as SD or XD memory, Random Access Memory (RAM),Static Random Access Memory (SRAM), Read-Only Memory (ROM), ElectricallyErasable Programmable Read-Only Memory (EEPROM), Programmable Read-OnlyMemory (PROM), magnetic memory, magnetic disk, and optical disk. Thevideo display device 100 may operate in linkage with a web storageperforming a storage function of the memory 150 in the Internet.

The memory 150 may include a content information storage unit 151, anenhanced service storage unit 152, a logo storage unit 153, a settinginformation storage unit 154, a bookmark storage unit 155, a userinformation storage unit 156, and a usage information storage unit 157.

The content information storage unit 151 stores a plurality of contentinformation corresponding to a plurality of characteristic information.

The enhanced service storage unit 152 may store a plurality of enhancedservices corresponding to a plurality of characteristic information or aplurality of enhanced services corresponding to a plurality of contentinformation.

The logo storage unit 153 stores a plurality of logos. Additionally, thelogo storage unit 153 may further store content provider identifierscorresponding to the plurality of logos or watermark server addressescorresponding to the plurality of logos.

The setting information storage unit 154 stores setting information forACR.

The bookmark storage unit 155 stores a plurality of bookmarks.

The user information storage unit 156 stores user information. The userinformation may include at least one of at least one account informationfor at least one service, regional information, family memberinformation, preferred genre information, video display deviceinformation, and a usage information range. The at least one accountinformation may include account information for a usage informationmeasuring server and account information of social network service suchas Twitter and Facebook. The regional information may include addressinformation and zip codes. The family member information may include thenumber of family members, each member's age, each member's sex, eachmember's religion, and each member's job. The preferred genreinformation may be set with at least one of sports, movie, drama,education, news, entertainment, and other genres. The video displaydevice information may include information such as the type,manufacturer, firmware version, resolution, model, OS, browser, storagedevice availability, storage device capacity, and network speed of avideo display device. Once the usage information range is set, the videodisplay device 100 collects and reports main audio and video contentwatching information and enhanced service usage information within theset range. The usage information range may be set in each virtualchannel. Additionally, the usage information measurement allowable rangemay be set over an entire physical channel.

The usage information providing unit 157 stores the main audio and videocontent watching information and the enhanced service usage information,which are collected by the video display device 100. Additionally, thevideo display device 100 analyzes a service usage pattern on the basisof the collected main audio and video content watching information andenhanced service usage information, and stores the analyzed serviceusage pattern in the usage information storage unit 157.

The enhanced service management unit 130 may obtain the contentinformation of the main audio and video content from the fingerprintserver 22 or the content information storage unit 151. If there is nocontent information or sufficient content information, which correspondsto the extracted characteristic information, in the content informationstorage unit 151, the enhanced service management unit 130 may receiveadditional content information through the data transmitting/receivingunit 141. Moreover, the enhanced service management unit 130 may updatethe content information continuously.

The enhanced service management unit 130 may obtain available enhancedservice from the enhanced service providing server 50 or the enhancedservice storage unit 153. If there is no enhanced service or sufficientenhanced service in the enhanced service storage unit 153, the enhancedservice management unit 130 may update enhanced service through the datatransmitting/receiving unit 141. Moreover, the enhanced servicemanagement unit 130 may update the enhanced service continuously.

The enhanced service management unit 130 may extracts a logo from themain audio and video content, and then, may make a query to the logostorage unit 155 to obtain a content provider identifier or watermarkserver address, which is corresponds to the extracted logo. If there isno logo or a sufficient logo, which corresponds to the extracted logo,in the logo storage unit 155, the enhanced service management unit 130may receive an additional logo through the data transmitting/receivingunit 141. Moreover, the enhanced service management unit 130 may updatethe logo continuously.

The enhanced service management unit 130 may compare the logo extractedfrom the main audio and video content with the plurality of logos in thelogo storage unit 155 through various methods. The various methods mayreduce the load of the comparison operation.

For example, the enhanced service management unit 130 may perform thecomparison on the basis of color characteristics. That is, the enhancedservice management unit 130 may compare the color characteristic of theextracted logo with the color characteristics of the logos in the logostorage unit 155 to determine whether they are identical or not.

Moreover, the enhanced service management unit 130 may perform thecomparison on the basis of character recognition. That is, the enhancedservice management unit 130 may compare the character recognized fromthe extracted logo with the characters recognized from the logos in thelogo storage unit 155 to determine whether they are identical or not.

Furthermore, the enhanced service management unit 130 may perform thecomparison on the basis of the contour of the logo. That is, theenhanced service management unit 130 may compare the contour of theextracted logo with the contours of the logos in the logo storage unit155 to determine whether they are identical or not.

Then, referring to FIGS. 11 and 12, a method of synchronizing a playbacktime of a main audio and video content with a playback time of anenhanced service according to an embodiment will be described.

FIG. 11 is a flowchart illustrating a method of synchronizing a playbacktime of a main audio and video content with a playback time of anenhanced service according to an embodiment.

Enhanced service information may include a start time of an enhancedservice. At this point, the video display device 100 may need to startthe enhanced service at the start time. However, since the video displaydevice 100 receives a signal transmitting an uncompressed main audio andvideo content with no time stamp, the reference time of a plying time ofthe main audio and video content is different from that of a start timeof the enhanced service. Although the video display device 100 receivesa main audio and video content having time information, the referencetime of a plying time of the main audio and video content may bedifferent from that of a start time of the enhanced service, likerebroadcasting. Accordingly, the video display device 100 may need tosynchronize the reference time of the main audio and video content withthat of the enhanced service. Especially, the video display device 100may need to synchronize the playback time of the main audio and videocontent with the start time of the enhanced service.

First, the enhanced service management unit 130 extracts a certainsection of a main audio and video content in operation S801. The sectionof the main audio and video content may include at least one of somevideo frames or a certain audio section of the main audio and videocontent. Time that the enhanced service management unit 130 extracts thesection of the main audio and video content is designated as Tn.

The enhanced service management unit 130 obtains content information ofa main audio and video content on the basis of the extracted section. Inmore detail, the enhanced service management unit 130 decodesinformation encoded with invisible watermark in the extracted section toobtain content information. Additionally, the enhanced servicemanagement unit 130 may extract characteristic information in theextracted section, and obtain the content information of the main audioand video content from the fingerprint server 22 or the contentinformation storage unit 151 on the basis of the extractedcharacteristic information. Time that the enhanced service managementunit 130 obtains the content information is designated as Tm.

Moreover, the content information includes a start time Ts of theextracted section. After the content information acquisition time Tm,the enhanced service management unit 130 synchronizes the playback timeof the main audio and video content with the start time of the enhancedservice on the biases of Ts, Tin, and Tn. In more detail, the enhancedservice management unit 130 regards the content information acquisitiontime Tm as a time Tp calculated by the following Equation 1.

Tp=Ts+(Tm−Tn)  [Equation 1]

Additionally, the enhanced service management unit 130 regards a time ofwhen Tx elapses after the content information acquisition time as Tp+Tx.

Then, the enhanced service management unit 130 obtains an enhancedservice and its start time Ta on the obtained content information inoperation S807.

If the synchronized playback time of the main audio and video content isidentical to the start time Ta of the enhanced service, the enhancedservice management unit 130 starts the obtained enhanced service inoperation S809. In more detail, the enhanced service management unit 130may start the enhanced service when the following Equation 2 issatisfied.

Tp+Tx=Ta  [Equation 2]

FIG. 12 is a conceptual diagram illustrating a method of synchronizing aplayback time of a main audio and video content with a playback time ofan enhanced service according to an embodiment.

As shown in FIG. 12, the video display device 100 extracts an AV sampleduring a system time Tn.

The video display device 100 extracts characteristic information fromthe extracted AV sample, and transmits a query including the extractedcharacteristic information to the fingerprint server 22 to receive aquery result. The video display device 100 confirms whether a start timeTs of the extracted AV sample corresponds to 11000 ms at Tm by parsingthe query result.

Accordingly, the video display device 100 regards the time of when thestart time of the extracted AV sample is confirmed according to Equation1 as Ts+(Tm−Tn), so that, after that, the playback time of the mainaudio and video content may be synchronized with the start time of theenhanced service.

Next, an ACR application will be described according to variousembodiments with reference to FIGS. 13 to 14.

The ACR application may include an auto playing application, a contentprovider application, and a video display manufacturer application.

FIG. 13 is a screen when an auto playing application is executedaccording to an embodiment.

The auto playing application automatically plays an enhanced serviceeven when a user makes no action. In this sense, the auto playingapplication is designated as a lean back application in thisspecification.

The auto playing application automatically plays an enhanced servicesuch as scene related information that a content provider orbroadcasting station provides during playing of the main audio and videocontent according to a playback time of the main audio and videocontent, without a user's input. That is, a content provider orbroadcasting station entirely controls the auto playing application. Forexample, when an auto playing application relates to a specific channelor a specific content, as the video display device 100 enters a channelor content relating to the auto playing application, the video displaydevice 100 may automatically execute a corresponding auto playingapplication without a user input. Then, when the video display device100 becomes out of a channel or content relating to an auto playingapplication, it may automatically terminate a corresponding auto playingapplication without a user input. In this sense, an auto playingapplication will be referred to as a channel-dependent application or acontent-dependent application in this specification.

As shown in FIG. 13, once the auto playing application is executed, thescreen 200 may include a sub screen 210 where a reduced main audio andvideo content is played and a sub screen 220 where additionalinformation is displayed. The sub screen 200 where additionalinformation is displayed may be spatially separated from the sub screen210 where a main audio and video content is played.

Moreover, when an auto playing application is executed, an area whereadditional information is displayed may be semi-transparent. In thiscase, the area where additional information is displayed may overlay onan area where a main audio and video content played.

FIG. 14 is a screen when a content provider application is executedaccording to an embodiment.

In this specification, the content provider application is designated asa full interactive application.

As shown in FIG. 14 (A), the video display device 100 downloads andstores a plurality of applications. FIG. 14 (A) illustrates icons 230 ofthe applications. A content provider application among the plurality ofapplications is an application created and distributed by a contentprovider, and thus, may be under control of the content provider.

A user of the video display device 100 may download a content providerapplication from an application store and determines whether to executethe downloaded content provider application.

FIG. 14 (B) is a screen when a content provider application is executed.As shown in FIG. 14 (B), the screen 200 may include the sub screen 210where a reduced main audio and video content is played and the subscreen 220 where additional information is displayed.

Hereinafter, a video display device manufacturer application will bedescribed according to an embodiment with reference to FIGS. 23 to 50.

The video display device manufacturer application is created andcontrolled by a video display device manufacturer, and may be stored inadvance when the video display device 100 is manufactured. That is, thevideo display device manufacturer application may provide the sameservice and user interface regardless of a content provider or content.

The video display device manufacturer application may include an areawhere an enhanced service that a content provider provides is displayedand an area where an enhanced service that a video display devicemanufacturer provides is displayed, separately. A user of the videodisplay device 100 may determine whether to execute the video displaydevice manufacturer application.

The video display device manufacturer application may have two types.

The video display device manufacturer application of the first type hasa resizing user interface. The video display device manufacturerapplication having the resizing user interface reduces the size of anarea where a main audio and video content is displayed to display allthe main audio and video content in a reduced area. Additionally, thevideo display device manufacturer application having the resizing userinterface reshapes an area where an application is displayed with an Lor inverse-L shaped structure.

The video display device manufacturer application of the second type hasan overlay user interface. The video display device manufacturerapplication having the overlay user interface maintains the size of anarea where a main audio and video content is displayed, and overlaps anarea where an application is displayed on the main audio and videocontent. Since the area where an application is displayed covers themain audio and video content, an application area may besemi-transparent.

Like this, a video display device manufacturer application is notdependent on any single contents provider or any single channel.Although it is described that such an application is manufactured by avideo display device manufacturer, the present invention is not limitedthereto. An application not dependent on any single contents provider orany single channel is not executed by a channel change or is notterminated by a channel change. For convenience of description in thisspecification, an application not dependent on any single contentprovider or any single channel may be referred to as achannel-independent application, a video display device manufacturerapplication or a Live+ app.

Then, an on/off control of an ACR function will be described accordingto an embodiment with reference to FIGS. 15 and 16.

FIG. 15 is a view of a user interface for controlling an ACR functionaccording to an embodiment.

As shown in FIG. 15, the play controlling unit 113 displays an ACRfunction setting window. A user may activate or deactivate the ACRfunction through the ACR function setting window.

When the ACR function is activated, a user may select an ACR mode. TheACR mode may be set with one of an auto playing application mode, acontent provider application mode, and a video display manufacturerapplication mode.

When the ACR function mode is set in the above manner, the playcontrolling unit 113 stores setting information on a setting informationstorage unit 154.

FIG. 16 is a flowchart illustrating a method of operating a videodisplay device according to the embodiment.

The enhanced service management unit 130 confirms whether the ACRfunction is turned on in operation S1901.

If the ACR function is deactivated, the enhanced service management unit130 does not perform a content information obtaining procedure any more.

If the ACR function is activated, the enhanced service management unit130 extracts a certain section of a main audio and video content inoperation S1902.

The enhanced service management unit 130 obtains content information ofthe main audio and video content on the basis of the extracted sectionin operation S1903.

The enhanced service management unit 130 synchronizes the reference timeof the main audio and video content with the reference time of analternative advertisement in operation S1905.

The enhanced service management unit 130 obtains an enhanced service onthe basis of the obtained content information in operation S1907. Theenhanced service management unit 130 obtains enhanced serviceinformation on the basis of the content information, and then obtains anenhanced service on the basis of the enhanced service information.

Then, the enhanced service management unit 130 confirms the ACR mode inoperation S1909.

The enhanced service management unit 130 plays the enhanced serviceaccording to the ACR mode in operation S1911.

In more detail, if the ACR mode is an automatic play application mode,the enhanced service management unit 130 executes an automatic playapplication, and the executed automatic play application plays theobtained enhanced service.

If the ACR mode is the content provider application mode, the enhancedservice management unit 130 executes a content provider applicationcorresponding to the obtained content information in background. Theenhanced service management unit 130 displays a content providerapplication when receiving a user input corresponding to an enhancedservice use. This content provider application may play the obtainedenhanced service. Instead that the content provider application isautomatically executed in background, it may be executed by a userinput.

If the ACR mode is the video display device manufacturer applicationmode, the enhanced service management unit 130 executes its videodisplay device manufacturer application in background. The enhancedservice management unit 130 displays the video display devicemanufacturer application when receiving a user input corresponding to anenhanced service use. This video display device manufacturer applicationmay play the obtained enhanced service. Instead that the video displaydevice manufacturer application is automatically executed in background,it may be executed by a user input.

Next, third party enhanced service will be described according to anembodiment of the present invention with reference to FIGS. 17 to 42.

FIG. 17 is a view illustrating a network topology according to anembodiment of the present invention.

As shown in FIG. 17, the network topology includes a contents providingserver 10 such as a Broadcaster/CP Backend server, a third party server90 such as the enhanced service providing server 50, a contentsrecognition service providing server 20 such as an ACR server, and avideo display device 100 such as an ACR Smart TV.

The video display device 100 may obtain broadcaster enhanced servicefrom the contents providing server 10 by using a channel-dependentapplication or a Leanback app such as a contents-dependent application,and may obtain third party enhanced service from the third party server90 by using a Live+ app such as a channel-independent application.

The Leanback app such as a channel-dependent application is anapplication that a broadcaster or a CP provides, and also provides aservice synchronized by a program unit or a scene unit. Since abroadcaster is a subject to produce a program, it knows well which sceneis in a program. Thus, the broadcaster may display the most suitablecontents for a specific scene. Once the video display device 100recognizes a program, it downloads a Leanback app from the contentsproviding server 10 of the corresponding program, and then,automatically executes the downloaded Leanback app. However, when thevideo display device 100 changes the current channel to another channelunrelated to the executed Leanback app, it automatically terminates thecorresponding Leanback app, and if there is a Leanback app of thechanged channel, it executes the corresponding Leanback app.

A Live+ app is an application that a third party provides (for example,a manufacturer), is pre-installed in a TV, and may be executed alwaysregardless of a channel or program that a user currently watches, unlikethe Leanback app. In relation to an ACR architecture that the presentinvention suggests, a Live+ app displays program related enhancedinformation and contents synchronized by a scene unit. The Live+ app hasa consistent user interface and is always executed regardless of aprogram. The Live+ app may provide various services by using programrelated metadata even when a broadcaster does not provide programrelated additional information. For example, the Live+ app may providebasic information on a program and recommendation on movies in which aprogram actor appears. Since the Live+ app has an area assigned to abroadcaster/CP, it may display corresponding program related enhancementdata on a corresponding area when the broadcaster provides programrelated additional information. For example, enhancement data that abroadcaster provides may include the replay of the previous broadcast ofthe program that a user currently watches and scene synchronizedcontents (for example, product purchase, place reservation, music sourcepurchase and download, and product ad).

The video display device 100 includes an ACR module to extract asignature from uncompressed audio and video content and queries thecontents recognition service providing server 20 through the extractedsignature in order to recognize a program. Also, the video displaydevice 100 drives a program related application on the basis of therecognized program. The Leanback app may be downloaded after the programrecognition and then executed. The Live+ app may be pre-installed by amanufacturer when a TV is released, or may be downloaded from an appstore and installed.

The contents recognition service providing server 20 stores thesignatures of audio and/or video of a broadcasted program. The contentsrecognition service providing server 20 may extract and store thesignature of a real-time broadcast by using a signature capture system,and may receive an ACR DB including a signature corresponding to aspecific program from a broadcaster and then may store it. The ACR DB ofthe contents recognition service providing server 20 stores the contentID, signature, timestamp, and URL of a program. In this specification,two types of content IDs may be used.

The first type of content ID, as an ID that a broadcaster directlyassigns, is an identifier that uniquely identifies a program in abroadcaster. The broadcaster may assign the content ID to the program inits desired format. The content ID may be used when program relatedenhancement data are requested to the content providing server 10.

The second type of content ID, as a global unique content ID, is anidentifier that uniquely identifies a program nationally or globally.Since programs are transmitted through national broadcast channels,local broadcast channels, cable broadcasts, or satellite broadcasts, inorder to extract the signatures of all the programs, a plurality ofsignature capture equipment may be installed in a plurality of areas.Also, in order to uniquely identify and distinguish each programnationally or globally, a global unique content ID may be assigned toeach program. In order to map each program into a global unique contentID, a metadata database may be used. Since the metadata databaseincludes program schedule information in addition to metadata such asprogram related additional information, the signature delivered from acapture system may be mapped into a certain program by using themetadata database. The capture system knows which area, channel, andtime the extracted signature relates to, and thus, when the metadatadatabase is searched with this information, a program may be found. Thecontents recognition service providing server 20 and the third partyserver 90 may share the metadata database, and may recognize a programthrough a global unique content ID by using the metadata database. Theformat of the global unique content ID may be designated by a thirdparty operating a server, or may be provided from the metadata database.

All services that a Live+ app provides are provided from the third partyserver 90. The third party server 90 may receive a global unique contentID that the contents recognition service providing server 20 delivers,from the video display device 100. The third party server 90 may extractprogram related metadata corresponding to the global unique content IDfrom the metadata database through the global unique content ID, and onthe basis of the extracted metadata, may provide program related variousservices to the video display device 100. Additionally, if there isadditional information on the recognized program that a broadcasterprovides, the third party server 90 accesses the broadcaster server inorder to receive program related enhancement data, and processes andprovides them to the video display device 100. The program relatedenhancement data that a broadcaster provides through a Live+ app havethe same contents as but a different user interface from the enhancementdata that a Leanback app. However, in another embodiment, according tobroadcasters, the program related enhancement data that a broadcasterprovides through a Live+ app may be different from those that a Leanbackapp provides.

There may be a plurality of contents providing servers 10 for aplurality of broadcasters or a plurality of CPs. The contents providingserver 10 has a program targeted Leanback app and program relatedenhancement data that the Leanback app outputs.

FIG. 18 is a view illustrating a Live+ app according to variousembodiments of the present invention.

The Live+ app may resize main AV media content in order to display it onthe upper left area of the screen, may display program relatedadditional information on the right area of the screen, and may displaya program related ad on the bottom area of the screen. The Live+ app maydisplay a plurality of tabs on the top area of the contents area in theright area. The plurality of tabs may include a program tab, arecommends tab, a talk tab, and a plus tap.

FIG. 19 is a view illustrating a method of executing a Live+ appaccording to various embodiments of the present invention.

The video display device 100 may display a home screen or a My Appsscreen. The My Apps screen is a screen displaying a list of applicationinstalled in the video display device 100.

On receiving a user input selecting a Live+ app from a plurality ofapplications shown on the home screen or the My Apps screen, the videodisplay device 100 may execute the Live+ app.

Additionally, while playing a broadcast, the video display device 100may display a graphic notifying that there is a Live+ app at apredetermined time interval through a timer, and when receiving a userinput relating to this graphic, may execute the Live+ app.

FIG. 20 is a view illustrating a tab function of a Live+ app accordingto an embodiment of the present invention.

The program tab provides basic information on a program that a viewercurrently watches and the replay service of the program. The videodisplay device 100 may fetch the program basic information from the MetaDB of the third party server 90. The video display device 100 may bringthe information on the program replay service from the contentsproviding server 10 that creates a corresponding program, through thethird party server 90.

The recommends tab may recommend another audio and video contentrelating to a current program, i.e. a movie or a video (for example,Youtube) or an app relating to the current program. The third partyserver 90 may determine a movie or a video for recommendation on thebasis of the title, genre, or cast of the current program, and then mayprocess the information about this to provide the recommendation tab ofthe Live+ app of the video display device 100.

The Talk tab provides a twitter article corresponding to the currentprogram related conversation content. The third party server 90 maysearches for the current program related twitter article on the basis ofthe title, genre, or cast of the current program, and then may processthe searched article to provide the Talk tab of the Live+ app of thevideo display device 100.

The Plus tab is an area where a broadcaster displays the program relateddata. The Plug tab may provide information received from the broadcasteras it is, or may provide information that is received from a broadcasterserver and processed by the third party server 90. There is no limit tothe type and number of data that the Plus tab provides. The Plus tab mayprovide various enhancement data such as product purchase, placereservation, music source purchase, and product ad. However, when theinformation from a broadcaster server is provided to the Plus tab as itis, the third party server 90 processes the information from thebroadcaster server to be fit for the position and size of the contentsarea of the Live+ app, and then provides the processed information tothe video display device 100

FIG. 21 is a view illustrating an architecture of the third party server90 according to an embodiment of the present invention.

As shown in FIG. 21, the third party server 90 includes an ApplicationLayer, a Service Component Layer, a Back-end System Layer, and a DataComponent Layer.

The application layer may communicate with various devices including thevideo display device 100 through a delivery interface. The applicationlayer may include an AppStore component. The AppStore component maysupport an interface through which a developer registers an app, andalso may support an interface through which a general user searches for,downloads, and installs an app. Additionally, the Live+ app may receiveservices such as ad, search, and recommendation in the third part server90 through the AppStore component.

A service component layer may communicate with the application layerthrough a service integration interface. The service component layerincludes an ad service component, an integration search component, arecommendation component, and an ACR component. The service componentlayer receives the processed data from the back-end system layer anddelivers them to the application layer. The ACR component may obtainprogram related basic additional information through the metadatacomponent of the back-end system layer by using the received Content ID,Global Content ID, timestamp, and broadcaster server URL from the videodisplay device 100. Additionally, the ACR component may deliver theinformation on a program title and actors obtained from the metadata tothe integration search component in order to a program related searchresult.

The back-end system layer reads original data from the data componentlayer and processes and delivers the data to the service componentlayer. The back-end system layer may include a search engine securitycomponent, a DRAM component, a metadata component, an ad platformcomponent, and an ACR component.

The data component layer may have actual data. Here, the data componentlayer may include a file repository, an integrated DB, a user profilerepository, a user profile repository, a metadata repository, and an addata storage. The user profile repository may store information on aviewer or user's gender, age, region, income, family member, preferredgenre, preferred channel, and preferred program. The metadata repositorymay include information on content's title, episode, director, synopsis,casts, genre, channel, and air time. The third party server 90 mayreceive information on a content from a metadata provider or real-timebroadcast guide information (for example, in the case of ATSCterrestrial waves, PSIP), and then may store the received information inthe metadata repository.

FIG. 22 is a ladder diagram illustrating an operation of a video displaydevice for an automatic execution application according to an embodimentof the present invention.

First, the video display device 100 extracts characteristic informationfrom some frames of the main audio and video content or a section ofaudio samples of the main audio and video content, and transmits an ACRquery request message including the extracted characteristic informationto a fingerprint server 22 corresponding to an ACR server in operationS2105.

The fingerprint server 22 queries characteristic information in the ACRquery request message from an ACR DB to identify a program correspondingto the characteristic information, and provides an ACR query responsemessage including contents information on the identified program to thevideo display device 100 in operation S2107. At this point, the ACRquery response message may include information on whether programrecognition is successful, contents information, and the URL of aLeanback app. As mentioned above, the content information may includethe content ID and timestamp of the recognized program. Here, the formatof the content ID may be a format that only a corresponding broadcasterrecognizes. Especially, this is the case that a broadcaster provides theACR DB of the corresponding program to the contents recognition serviceproviding server 20.

When the program identification is successful and the Leanback app URLis valid, the video display device 100 requests a Leanback app through acorresponding Leanback app URL in operation S2109.

The contents providing server 10 provides the Leanback app that thevideo display device 100 requests to the video display device 100 inoperation S2111. Consequently, the video display device 100 may downloadthe Leanback app.

When the video display device 100 successfully downloads the Leanbackapp, it automatically executes a corresponding Leanback app in operationS2113.

The Leanback app executed in the video display device 100 obtains acontent ID and a timestamp periodically from an ACR engine in the videodisplay device 100, and transmits an enhancement data request messageincluding the obtained content ID and timestamp to the contentsproviding server 10 in operation S2115, and then requests theenhancement data relating to a scene corresponding to the content ID andthe timestamp.

The contents providing server 10 delivers the requested enhancement datato the Leanback app. Since the Leanback app is an application that abroadcaster provides, an interface of the contents providing server 10of one broadcaster may be different from that of the contents providingserver 10 of another broadcaster. When there is scene related additionalinformation, the Leanback app may perform operations S2113 and S2117repeatedly.

FIG. 23 is a ladder diagram illustrating an operation of a video displaydevice for a channel-independent application according to an embodimentof the present invention.

On receiving a user input to execute an installed Live+ app, the videodisplay device 100 executes the Live+ app in operation S2201.

When an ACR function is turned off, the Live+ app executed in the videodisplay device 100 turns on the ACR module in operation S2203.

The video display device 100 extracts characteristic information fromsome frames of the main audio and video content or a section of audiosamples of the main audio and video content, and transmits an ACR queryrequest message including the extracted characteristic information tothe ACR server corresponding to the fingerprint server 22 in operationS2205.

The fingerprint server 22 queries characteristic information in the ACRquery request message from an ACR DB to identify a program correspondingto the characteristic information, and provides an ACR query responsemessage including contents information on the identified program to thevideo display device 100 in operation S2207. At this point, the ACRquery response message may include information on whether programrecognition is successful and contents information. As mentioned above,the content information may include the global unique content ID andtimestamp of the recognized program.

The Live+ app executed in the video display device 100 receives a userinput for service type selection in operation S2209. At this point, theservice type may correspond to at least one of a program type, arecommendation type, a conversation type, and a plus type. For example,enhancement data corresponding to the selected service type may be areal-time news article relating to cast members and a servicerecommending a movie in which a program cast member appears as a leadingrole.

The Live+ app executed in the video display device 100 obtains contentsinformation on the currently recognized program from the ACR engine inthe video display device 100, and transmits a program related contentrequest message to the third party server 90 in operation S2211 in orderto request enhancement data corresponding to the selected service type.At this point, the program related content request message may includecontent information on the currently recognized program and informationon the selected service type.

The Live+ app may transmit an HTTP based request to the third partyserver 90 through ajax call. Parameters delivered at this point mayinclude a service type, a Content ID, a Global Content ID, a timestamp,and a broadcaster server URL. The HTTP request may deliver a HTTP GETbased parameter using the following syntax.

[HTTP GET syntax] ?service=<service name>&contentId=<Content ID>&globalContentId=<Global Content ID>&ts=<timestamp> [&url=< broadcasterserver URL>]

<service name> is the name of a service that a Live+ app requests. Inthis embodiment, the following <service name> may be used.

-   -   “getProgram”: is a service name for receiving program related        basic information and previous episodes, and is used for a        program tab of a Live+ app.    -   “getRecommends”: is a service name for receiving data of program        related recommendation (VOD recommendation and App        recommendation), and is used for a recommends tab of a Live+        app.    -   “getTalk”: is a service name for receiving program related        article/twitter, and is used for a talk tab of a Live+ app.    -   “getPlus”: is a service name for receiving program related        enhancement data that a broadcaster provides, and is used for a        plus tab of a Live+ app.

<Content ID> is a value that a broadcaster delivers to the contentsrecognition service providing server 20, and is a content ID that isinternally used by a broadcaster in order to uniquely identify aprogram.

<Global Content ID> is used for uniquely and globally identifying aprogram, and for this, the contents recognition service providing server20 and the third party server 90 may need to use the same Meta DB. TheGlobal Content ID may follow the content id format of the Meta DB.

<timestamp> notifies the current watching time of the recognized programand is a value delivered from the contents recognition service providingserver 20.

<broadcaster server URL> is delivered when a broadcaster providesprogram related enhancement data, and with this value, the third partyserver 90 may access a broadcaster server.

In order to locate enhancement data of a program corresponding to thedelivered Global Content ID, the third party server 90 searches themetadata repository in operation S2213. The metadata repository returnsa search result on the enhancement data of a program corresponding tothe delivered Global Content ID, to the third party server 90.

The third party server 90 processes the received program enhancementdata from the metadata repository and provides the processed enhancementdata to the Live+ app executed in the video display device 100 inoperation S2221. The third party server 90 may transmit the processedenhancement data in an HTTP based response through ajax call.

FIG. 24 is a ladder diagram illustrating an operation of a video displaydevice for a channel-independent application according to anotherembodiment of the present invention.

On receiving a user input to execute an installed Live+ app, the videodisplay device 100 executes the Live+ app in operation S2301.

When an ACR function is turned off the Live+ app executed in the videodisplay device 100 turns on the ACR module in operation S2203.

The ACR engine of the video display device 100 extracts characteristicinformation from some flames of the main audio and video content or asection of audio samples of the main audio and video content, andtransmits an ACR query request message including the extractedcharacteristic information to the ACR server corresponding to thefingerprint server 22 in operation S2305.

The fingerprint server 22 queries characteristic information in the ACRquery request message from an ACR DB to identify a program correspondingto the characteristic information, and provides an ACR query responsemessage including contents information on the identified program to theACR engine of the video display device 100 in operation S2307. At thispoint, the ACR query response message may include information on whetherprogram recognition is successful, contents information, and the URL ofthe contents providing server 10. As mentioned above, the contentinformation may include the global unique content ID of the recognizedprogram and the content ID and timestamp that the broadcaster of therecognized program identifies. In another embodiment, the video displaydevice 100 may have the URL of the contents providing server 10 inadvance, not obtaining the URL from the ACR query response message.

The Live+ app executed in the video display device 100 receives a userinput for service type selection in operation S2309. At this point, theservice type may correspond to at least one of a program type, arecommendation type, a conversation type, and a plus type. For example,enhancement data corresponding to the selected service type may be areal-time news article relating to cast members and a servicerecommending a movie in which a program cast member appears as a leadingrole.

The Live+ app executed in the video display device 100 obtains contentsinformation on the currently recognized program from the ACR engine inthe video display device 100, and transmits a program related contentrequest message to the third party server 90 in operation S2311 in orderto request enhancement data corresponding to the selected service type.At this point, the program related content request message may includecontent information on the currently recognized program, information onthe selected service type, and the URL of the broadcaster contentsproviding server 10.

In order to locate enhancement data of a program corresponding to thedelivered Global Content ID, the third party server 90 searches themetadata repository in operation S2313. The metadata repository returnsa search result on the enhancement data of a program corresponding tothe delivered Global Content ID, to the third party server 90.

The third party server 90 accesses the delivered URL of the broadcastercontents providing server 10, and transmits a metadata request messageincluding the delivered broadcaster content ID and timestamp to thecontents providing server 10 in operation 2317 in order to request thecurrent scene related enhancement data to the contents providing server10.

The third party server 90 receives the current scene related enhancementdata from the contents providing server 10 in operation S2319.

The third party server 90 processes one or a combination of the receivedprogram enhancement data from the metadata repository and the receivedprogram enhancement data from the contents providing server 10 andprovides the processed enhancement data to the Live+ app executed in thevideo display device 100 in operation S2321.

FIG. 25 is a conceptual diagram illustrating a video display deviceaccording to an embodiment of the present invention.

As shown in FIG. 25, the video display device 100 includes anAudio/Video driver 601, an ACR engine 603, an ACR middleware 605, an ACRuser interface 607, and an ACR Plugin 609.

The audio/video driver 601 captures an audio/vide signal from anexternal input of a TV and delivers it to the ACR engine 603. Theaudio/video driver 601 may provide API to allow the ACR engine 603 toaccess an audio/video buffer. The audio/video driver 601 may alsoprovide characteristic information on audio/video inputted from anexternal input such as HDMI. For example, the audio/video driver 601 mayprovide information such as a sample depth, a sampling rate, the numberof channels (mono/stereo

), the number of samples, and a sample time, in the case of audio. Inthe case of video, the audio/video driver 601 may provide informationsuch as a video format, and the width, height, stride, stream time,input type (HDMI1, HDMI2, composite, component, etc), and the number offrames per second of video.

The ACR engine 603 may be executed as an additional process, and mayrecognize a program by using the above mentioned various methods. TheACR engine 603 extracts a signature from an audio/video sample, anddelivers the extracted signature to the ACR server to recognize content.As described above, the ACR engine 603 may call API to access anaudio/video buffer depending on the implementation of a platform, andmay deliver the audio/video stream captured by the audio/video driver601 to an ACR process through a method such as a socket ACR engine 603sends an ACR query request in addition to the signature to the ACRserver, and receives an ACR query response including whether the contentrecognition is successful, a Content ID, a Global Content ID, atimestamp, a Leanback app URL, and a broadcaster server URL, from theACR server. The ACR engine 603 may deliver the result from the ACRserver to the ACR middleware 605.

The ACR middleware 605 is a module corresponding to the middleware of aDTV process, and processes an ACR control and an ACR query response fromthe ACR engine 603. The ACR middleware 605 controls ACR by executing orterminating an ACR process or starting or stopping the ACR engine 603.Additionally, the ACR middleware 605 stores values such as a Content ID,a Global Content ID, and a timestamp by parsing the ACR query response.The ACR middleware 605 may provide API to transmit the stored value toan ACR UI or may transmit the stored value to the ACR user interface 607through a message queue and a global variable. Additionally, in order todeliver data such as a Content ID, a Global Content ID, and a timestampto an ACR application such as a Live+ app or a Leanback app, the ACRmiddleware 605 may deliver the data to a web browser throughInter-Process Communication (IPC) of a shared memory and socket.

The ACR user interface 607 may display a UI to a user for ACR control ofACR On/Off, or may display information on the name and time of arecognized program to a user through a status bar UI of a TV.

When the ACR middleware 605 delivers data such as a Content ID, a GlobalContent ID, and a timestamp, which are to be delivered to an ACRapplication, to a browser, the ACR plugin 609 receives the data. The ACRplugin 609 may deliver a corresponding value to an ACR applicationthrough a Plugin Object Interface. An interface of the ACR plugin 609according to an embodiment is shown below.

Name Property Readonly ACRMetadata metadata Readonly String backendURLMethod backendURL getACRMetadata( ) String getBackendURL( ) VoidNotifyACRAppLaunched(Boolean bApp, String url, String desc) VoidSetAppInfo(String state, String url) Event functiononContentChanged(String contentId) function onMediaTimeUpdated(NumbermediaTime) function onAppShow(String state) function onAppHide( )

As mentioned above, the metadata provide basic metadata on a recognizedprogram.

backendURL represents the URL of a broadcaster/CP server.

getACRMetadata( ) is a function for obtaining basic metadata on arecognized program and returns the same value as metadata Property.

getBackendURL( ) is a function for returning the URL of a broadcaster/CPserver.

NotifyACRAppLaunched( ) is a function for notifying an application of a2nd screen when the 2nd screen such as a mobile terminal and a TV arepaired and an ACR application executes a specific URL or app on a TV.

SetAppInfo( ) is a function for notifying the current state of an ACRapplication and the app URL of a 2nd screen version of a correspondingapplication.

onContentChanged( ) is a callback function called when a recognizedprogram is changed.

onMediaTimeUpdated( ) is a callback function called each time atimestamp is changed through ACR.

onAppShow( ) is a callback function called when an application in a 2ndscreen moves to a TV, with the TV and 2nd screen paired.

onAppHide( ) is a callback function called when an application in a TVmoves to a 2nd screen, with the TV and 2nd screen paired.

FIG. 26 is a block diagram illustrating a fingerprint based networktopology according to another embodiment.

As shown in FIG. 26, the network topology further includes a videofingerprint server 23 and an audio fingerprint server 24.

The video fingerprint server 23 as shown in FIG. 26 does not edit a mainaudio and video content, but extracts visual video characteristicinformation from the main audio and video content and stores theextracted visual video characteristic information. Then, upon receivingvideo characteristic information from the video display device 100, thevideo fingerprint server 23 provides an identifier and time informationof an audio and video content corresponding to the received videocharacteristic information.

The audio fingerprint server 24 as shown in FIG. 26 does not edit themain audio and video content, but extracts auditory audio characteristicinformation from the main audio and video content and stores theextracted auditory audio characteristic information. Then, uponreceiving audio characteristic information from the video display device100, the audio fingerprint server 24 provides an identifier and timeinformation of an audio and video content corresponding to the receivedaudio characteristic information.

Here, when the video characteristic information and the audiocharacteristic information are simultaneously used, the video displaydevice 100 as shown in FIG. 26 may separately access the videofingerprint server 23 or the audio fingerprint server 24.

FIG. 27 is a block diagram illustrating a fingerprint based networktopology according to another embodiment.

As illustrated in FIG. 27, the network topology further includes afingerprint server 22 that stores video characteristic information andaudio characteristic information.

The fingerprint server 22 as shown in FIG. 27 does not edit a main audioand video content, but extracts visual video characteristic informationor auditory audio characteristic information from the main audio andvideo content and stores the extracted visual video characteristicinformation or auditory audio characteristic information. Then, uponreceiving video characteristic information from the video display device100, the fingerprint server 22 provides an identifier and timeinformation of an audio and video content corresponding to the receivedvideo characteristic information, or, upon receiving audiocharacteristic information from the video display device 100, thefingerprint server 22 provides an identifier and time information of anaudio and video content corresponding to the received audiocharacteristic information.

Here, even though the video characteristic information and the audiocharacteristic information are simultaneously used, the video displaydevice 100 as illustrated in FIG. 27 may access the single fingerprintserver 22.

FIG. 28 is a block diagram illustrating a structure of a fingerprintbased video display device according to another embodiment.

Referring to FIG. 28, a tuner/demodulator/demux 301 restores an analogaudio video signal from a received terrestrial broadcast signal (RF),and transmits the restored analog audio video signal to an ADC/AVdecoder 303.

The ADC/AV decoder 303 converts an analog audio video signal, an analogvideo signal or an analog audio signal into a digital analog audio videosignal, a digital video signal or a digital audio signal, and thendecodes the digital signals.

The analog audio video signal may be, for example, the analog audiovideo signal restored by the tuner/demodulator/demux 301. The analogvideo signal may be, for example, an analog video signal(Composite/S-video/Component/RGB) transmitted from an external inputdevice. The analog audio signal may be, for example, an analog audiosignal (Audio) transmitted from the external input device.

A digital signal processor 305 processes a digital audio video signal, adigital video signal or a digital audio signal.

The digital signal processor 305 may process the digital audio videosignal, the digital video signal or the digital audio signal decoded bythe ADC/AV decoder 303. Furthermore, the digital signal processor 305may process at least one of a digital audio video signal, a digitalvideo signal and a digital audio signal transmitted from an externalinput device through a high definition multimedia interface (HDMI) or adigital video interactive (DVI).

The digital signal processor 305 captures an audio stream to generate anaudio sample, and then transmits the generated audio sample to an audioFP module 307. The digital signal processor 305 captures a video streamto generate a video sample, and then transmits the generated videosample to a video FP module 309. The digital signal processor 305transmits screen information to a display 321.

The audio FP module 307 analyzes and processes the audio sample toextract audio characteristic information (Audio Signature).

The video FP module 309 analyzes and processes the video sample toextract video characteristic information (Video Signature).

A network I/F 311 makes a query on content information of a main audioand video content that is currently played while transmitting the audiocharacteristic information (Audio Signature) to the above-describedaudio fingerprint server 24 or fingerprint server 22 capable ofproviding an audio fingerprint service, and receives a result of thequery from the audio fingerprint server 24 or the fingerprint server 22(Query/Response).

A network I/F 313 makes a query on content information of a main audioand video content that is currently played while transmitting the videocharacteristic information (Video Signature) to the above-describedvideo fingerprint server 23 or fingerprint server 22 capable ofproviding a video fingerprint service, and receives a result of thequery from the video fingerprint server 23 or the fingerprint server 22(Query/Response).

The audio query result parser 315 analyzes a query result transmittedfrom the audio fingerprint server 24 or the fingerprint server 22capable of providing the audio fingerprint service, and extractsmetadata required by the video display device 100.

The video query result parser 317 analyzes a query result transmittedfrom the video fingerprint server 23 or the fingerprint server 22capable of providing the video fingerprint service, and extractsmetadata required by the video display device 100.

The metadata storage 319 stores the extracted metadata.

The display 321 displays scene related information on a screen.

FIG. 29 illustrates a user interface for controlling a fingerprintfunction according to an embodiment.

As shown in FIG. 29, the play controlling unit 113 displays afingerprint function setting window. A user may activate or deactivatethe fingerprint function through the fingerprint function settingwindow.

When the fingerprint function is activated, the user may select afingerprint setting mode. The fingerprint setting mode according to anembodiment may be set as one of a manual setting mode and an automaticsetting mode.

When the manual setting mode is activated, the user may select asignature mode. The signature mode according to an embodiment may be setas one of an audio signature mode, a video signature mode, and an audioand video signature mode.

Furthermore, when the manual setting mode is activated, the user mayselect a signature transmission period. The signature transmissionperiod may be changed according to a function of the video displaydevice 100, and the video display device 100 may automatically set thesignature transmission period. In the case where the video displaydevice 100 automatically sets the signature transmission period, thevideo display device 100 may set a predetermined default value as thesignature transmission period, or may set the signature transmissionperiod according to a state of an environment of a network where asignature is transmitted.

When the automatic setting mode is activated, the user may select asignature mode. The signature mode according to an embodiment may be setas one an audio preferred mode, a video preferred mode, and a concurrentaudio and video mode.

According to the audio preferred mode, a query including videocharacteristic information is transmitted when a result of a query onaudio characteristic information is a failure (Audio first, then VideoFingerprinting). According to the video preferred mode, a queryincluding audio characteristic information is transmitted when a resultof a query on video characteristic information is a failure (Videofirst, then Audio Fingerprinting). According to the concurrent audio andvideo mode, the query including the audio characteristic information andthe query including the video characteristic information areconcurrently transmitted (Fingerprinting concurrently). The signaturetransmission period at the signature mode according to an embodiment maybe automatically set by the video display device 100 as described above,but is not limited thereto.

When a mode of the fingerprint function is set as described above, theplay controlling unit 113 stores setting information in the settinginformation storage unit 154.

In the case where the query including the audio characteristicinformation and the query including the video characteristic informationare concurrently transmitted, the use of resources of the video displaydevice 100 may increase, degrading the performance of the device. Inorder to prevent the performance degradation of the video display device100, the signature transmission period may need to be changed accordingto a state of an environment of a network. Hereinafter, the change ofthe signature transmission period according to an embodiment will bedescribed with reference to FIGS. 30 and 31.

FIG. 30 is a graph illustrating a change of a fingerprint periodaccording to an embodiment.

Referring to FIG. 30, the video display device 100 sets a period of Afingerprint (A FP) and a period of B fingerprint (B FP) so that both theperiods are short when content is not recognized, and increases theperiod of the B fingerprint (B FP) when the content is recognized so asto reduce the use of resources of the video display device 100. Here,the A fingerprint (A FP) or the B fingerprint (B FP) may represent theaudio characteristic information or the video characteristicinformation, and the period of the A fingerprint (A FP) and the periodof the B fingerprint (B FP) may represent the signature transmissionperiod.

FIG. 31 is a graph illustrating a change of a fingerprint periodaccording to another embodiment.

Referring to FIG. 31, the video display device 100 may increase both theperiod of the A fingerprint (A FP) and the period of the B fingerprint(B FP) after recognition of content, and then may decrease only theperiod of the A fingerprint (A FP) if B fingerprinting fails. Here, theB fingerprinting may represent transmission of a query including theaudio characteristic information or the video characteristicinformation, and the failure of the B fingerprinting may represent afailure of a result of a query on the audio characteristic informationor the video characteristic information.

As a result, according to the change of the fingerprint period accordingto an embodiment, the video display device 100, to which a plurality offingerprinting operations are applied, reduces the use of resources forfailed fingerprinting and increases the use of resources for successfulfingerprinting, thereby improving the efficiency of recognizing content.

When a fingerprint is used to obtain content information, variousfactors may affect the use of the fingerprint, and various settings maybe applied according to situations. To this end, setting information forfingerprint based ACR may also be stored in the above-described storageunit 154. In the case where the video characteristic information and theaudio characteristic information are concurrently used based on thesetting information for ACR, a method of combining the videocharacteristic information and the audio characteristic information maybe derived. The setting formation for ACR may be obtained through userinput or detection of the video display device 100, or may be providedfrom the content recognition service providing server 20 such as thecontent providing server or the fingerprint server.

The play controlling unit 113 may operate based on the settinginformation for ACR stored in the setting information storage unit 154.Alternatively, the play controlling unit 113 may operate based oninformation that affects ACR operation, wherein the information isprovided from the outside, for example, the content providing server 10or the fingerprint server 22.

The setting information for ACR may include information on signal inputaccording to a type of an external input interface, information on aperiod of signature transmission, information on a sampling window thatis a video characteristic information extraction region, and informationon priority determination for the case where a query resultcorresponding to the audio characteristic information is different fromthat corresponding to the video characteristic information.

The information on the signal input according to the type of theexternal input interface may include information indicating whethersignals respectively inputted through a plurality of external inputinterfaces include audio signals and video signals and formatinformation on each of the audio signals and the video signals.

The video display device 100 may operate based on the informationindicating whether the signals respectively inputted through theplurality of external input interfaces include the audio signals and thevideo signals. For example, the video display device 100 may extract anduse only the video characteristic information from a signal receivedfrom a first external input interface, based on information indicatingthat the signal inputted through the first external input interface doesnot include an audio signal. Furthermore, the video display device 100may more rapidly perform content recognition operation using the formatinformation on each of the audio signals and the video signals stored inthe setting information storage unit 154.

The video display device 100 may collect and store, according to thetype of the external input interface, the information on the signalinput according to the type of the external input interface, and maystore information on a most recently inputted signal. Furthermore, inthe case where an external input device connected to the external inputinterface is identifiable, the video display device 100 may also storeinformation on signal input according to a type of each external inputdevice. For example, since the video display device 100 may obtaininformation on an external input device connected through an HDMI, thevideo display device 100 may store information on the external inputdevice according to a type of the external input device. In the casewhere information on an external input device is unobtainable, the videodisplay device 100 may store information on a most recently inputtedsignal.

The information on the period of signature transmission may representinformation indicating how frequently signature transmission isperformed, and may be set as a time period such as 5 seconds. Thesignature transmission period may include not only a basic signaturetransmission period but also a period of combining and using the videocharacteristic information and the audio characteristic information inthe case where the video characteristic information and the audiocharacteristic information are concurrently used. When the videocharacteristic information is used as a signature, a content recognitionspeed of the video display device 100 is high. On the contrary,consumption of resources such as CUP usage increases due to processingof a large amount of data. Therefore, the signature transmission periodmay be dynamically set in order to appropriately control the contentrecognition speed and the resource consumption of the video displaydevice 100, and information on the signature transmission period settingmay be stored as the setting information for ACR.

The information on the signature transmission period may include, forexample, information indicating that the video characteristicinformation is used or the video characteristic information and theaudio characteristic information are concurrently used for initialcontent recognition, information indicating that a transmission periodof the video characteristic information is gradually increased aftercontent recognition provided that it is confirmed that the same serviceor the same content is being viewed through the audio characteristicinformation, and information indicating that correct content recognitionis immediately attempted through the video characteristic information orthe video characteristic information and the audio characteristicinformation when a change of the service or the content that is beingviewed is detected through the audio characteristic information. Here,the transmission period of the video characteristic information may beincreased in such a manner that, for example, the period is 5 secondswithin 5 minutes from the time of viewing the same content, the periodis increased to 10 seconds within 10 minutes after a lapse of 5 minutesfrom the time of viewing the same content, the period is increased to 20seconds within 15 minutes after a lapse of 10 minutes from the time ofviewing the same content, and the period is increased two times as aviewing time increases by 5 minutes after a lapse of 15 minutes from thetime of viewing the same content. However, the transmission period ofthe video characteristic information is not limited thereto.

The information on the sampling window that is the video characteristicinformation extraction region may be designated according to GUIcharacteristic information on broadcast receiving devices that aredifferent from each other according to business operators. That is, whenthe query result corresponding to the video characteristic informationtransmitted from the video display device 100 is a failure, thefingerprint server 22 may analyze a pattern history of the video displaydevice 100 to estimate an operator of a broadcast receiving device ormultichannel video distributor, and may designate the sample window inconsideration of the GUI characteristic information on the broadcastreceiving device or the multichannel video distributor of the operator.Therefore, the video display device 100 may use different samplingwindows according to a type of the video display device 100 and a typeof the broadcast receiving device or the multichannel video distributor.The video display device 100 may store information on the samplingwindow as the setting information for ACR, and then may request a queryincluding the sampling window.

The information on the priority determination for the case where thequery result corresponding to the audio characteristic information isdifferent from that corresponding to the video characteristicinformation may include a method of pre-designating information to bedetermined as a priority or a method of calculating information to bedetermined as a priority through an algorithm. In particular, in thecase where the method of pre-designating one piece of information isdifferent from the method of determining priority information through analgorithm, the video display device 100 may determine any one type ofinformation as a signature based on the information on the prioritydetermination stored as the setting information for ACR.

FIG. 32 is a ladder diagram illustrating a data flow for contentrecognition in a fingerprint based network topology according to anembodiment.

The content providing server 10 such as a broadcaster/CP extracts afingerprint of content and determines an ACR query period in operationS2401.

The broadcaster/CP may establish an ACR DB for all programs possessed bythe broadcaster/CP, or may establish the ACR DB for a specific program.In the case where the ACR DB is established for a specific program, forexample, a popular program possessed by the broadcaster/CP, a servermaintaining cost may be reduced, and a time taken for making an inquiryfor program identification may be reduced.

The broadcaster/CP may extract a fingerprint of a program using a toolprovided by an ACR operator, may determine the ACR query period of theprogram, and then may store the ACR query period with the extractedfingerprint.

The ACR query period of the program may be determined according to acharacteristic of content.

For example, the ACR query period may be determined according to whetheran additional service related to the content exists. In the case wherethe additional service related to the service does not exist, the ACRquery period may be determined as a time at which the program isfinished. On the contrary, in the case where the additional servicerelated to the service exists, the ACR query period may be moreshortened. It there is no additional service related to the content, theACR query is transmitted only when the program is finished so as not totransmit an unnecessary ACR query, and thus, the cost may be reduced. Onthe contrary, if there is the additional service related to the content,the ACR query period is set to be short so as to provide the additionalservice synchronized with the program.

For another example, the ACR query period may be determined according toa purpose of the ACR query, such as audience measurement of abroadcaster. Therefore, even though there is no additional servicerelated to the content, the broadcaster may determine the ACR queryperiod as a short time, for example, 3 seconds, in order to perform theaudience measurement.

As described above, the content providing server 10 may differentlydetermine the ACR query period so that the ACR query period is suitablefor the purpose of using the ACR query of a broadcaster orcharacteristics of content, for each channel or each content.

Thereafter, the content providing server 10 transmits the extractedfingerprint and the determined ACR query period to the fingerprintserver 22 in operation S2403. Here, the content providing server 10 mapsa content ID for the program to the fingerprint in order to transmit thefingerprint.

The content providing server 10 may transmit the fingerprint and the ACRquery period before the program is broadcasted or at the same time asthe extraction of the fingerprint. For example, the broadcaster/CP maytransmit the fingerprint for a pre-produced program and the ACR queryperiod to the fingerprint server 22 before the program is broadcasted.Alternatively, in the case where a live program is broadcasted, thebroadcaster/CP may extract the fingerprint for the live program in realtime, and then may transmit the extracted fingerprint and the ACR queryperiod to the fingerprint server 22.

The fingerprint server 22 stores the received fingerprint and the ACRquery period in operation S2405. The fingerprint server 22 may store thecontent ID, the time stamp, the fingerprint and the ACR query period ofthe program in the ACR DB.

Thereafter, the video display device 100 extracts a fingerprint from anaudio sample of a partial frame or a partial interval of the main audioand video content, and transmits an ACR query request message includingthe extracted fingerprint to the fingerprint server 22 corresponding toan ACR server in operation S2407.

The fingerprint server 22 queries the fingerprint included in the ACRquery request message from the ACR DB to identify a programcorresponding to the fingerprint, and provides an ACR query responsemessage including a first ACR query period for the identified program tothe video display device 100 in operation S2409. Here, the ACR queryresponse message may include various types of content information inaddition to the first ACR query period. Hereinafter, examples of thecontent information on the program included in the ACR query responsemessage will be described with reference to FIG. 33 and Table 5.

FIG. 33 is an XML schema diagram of ACR-Resulttype containing a queryresult according to another embodiment.

As shown in FIG. 33, ACR-Resulttype containing a query result includesResultCode attributes and Content ID, NTPTimestamp, QueryInterval andConfidence elements.

For example, if the ResultCode attribute has a value of 200, this maymean that the query result is successful. If the ResultCode attributehas a value of 404, this may mean that the query result is unsuccessful.

The NTPTimestamp element may include at least one of a start time and anend time of a content section from which a fingerprint is extracted.

The QueryInterval element represents the ACR query period, and indicatesa transmission time of a next ACR query. The next ACR query time mayrepresent an absolute time or a relative time. The absolute ACR querytime may represent a time at which the next ACR query is to betransmitted, and the relative ACR query time may represent a wait timeuntil the next ACR query is transmitted.

The Confidence element represents the accuracy of content recognition.The Confidence element may have a value of one of 0 to 100 according toa degree of mapping. For example, when it is confirmed that afingerprint is correctly mapped to a program, the Confidence element mayhave a value of 100.

Table 5 illustrates an XML schema of ACR-ResultType containing the queryresult.

TABLE 5 <?xml version=“1.0” encoding=“UTF-8”?> <xs:schemaxmlns:xs=“http://www.w3.org/2001/XMLSchema”elementFormDefault=“qualified”attributeFormDefault=“ungualified”> <xs:complexTypename=“ACR-ResultType”> <xs:sequence> <xs:element name=“ContentID”type=“xs:anyURI”/> <xs:element name=“NTPTimestamp”type=“xs:unsignedLong”/> <xs:element name=“QueryInterval”type=“xs:unsignedInt” minOccurs=“0”/> <xs:element name=“Confidence”Type=“xs:unsignedInt” minOccurs=“0”/> <xs:element name=“##other”processContents=“skip” minOccurs=“0” maxOccurs=“unbounded”/></xs:sequence> <xs:attribute name=“ResultCode” type=“xs:string”use=“required”/> <xs:anyAttribute processContents=“skip”/></xs:complexType> </xs:schema>

Hereinafter, FIG. 32 is described again.

Thereafter, after a lapse of the time corresponding to the first ACRquery period, if the video display device 100 extracts a fingerprintfrom an audio sample of another partial frame or another partial sectionof the main audio and video content, and transmits an ACR query requestmessage including the extracted fingerprint to the fingerprint server 22corresponding to an ACR server in operation S2411, the fingerprintserver 22 queries the fingerprint included in the ACR query requestmessage from the ACR DB to identify a program corresponding to thefingerprint, and provides an ACR query response message including asecond ACR query period for the identified program to the video displaydevice 100 in operation S2413.

Furthermore, after a lapse of the time corresponding to the second ACRquery period, if the video display device 100 extracts a fingerprintfrom an audio sample of another partial frame or another partial sectionof the main audio and video content, and transmits an ACR query requestmessage including the extracted fingerprint to the fingerprint server 22corresponding to an ACR server in operation S2415, the fingerprintserver 22 queries the fingerprint included in the ACR query requestmessage from the ACR DB to identify a program corresponding to thefingerprint, and provides an ACR query response message including athird ACR query period for the identified program to the video displaydevice 100 in operation S2417.

Here, the first to third ACR query periods may be the same or differentfrom each other.

Hereinafter, a method of determining an ACR query period according to anembodiment will be described with reference to FIG. 34 and Tables 6 and7.

FIG. 34 is a diagram illustrating a fingerprint period of each of aplurality of video display devices included in a fingerprint basednetwork topology according to an embodiment.

Referring to FIG. 34, program A represents a one-hour program without arelated additional service, and program B represents a 30-minute programwith a related additional service. A broadcaster accordingly establishesan ACR DB for the program A and the program B, wherein an ACR queryfrequency for the program A is set to be minimum, whereas the ACR queryfrequency for the program B that is a popular program is set higher.

Referring to FIG. 34, when a channel is changed at a first time point t1to start playback of the program A, a first video display device(device 1) receives an ACR query response as illustrated in Table 6 fromthe fingerprint server 22.

TABLE 6 <ACR-Result ResultCode= “200” > <ContentID>ProgramA</ContentID><NTPTimestamp>900000</NTPTimestamp> <QueryInterval>2700</QueryInterval><Confidence>100</Confidence> </ACR-Result>

Referring to Table 6, QueryInterval represents a relative ACR query timeto wait until a next ACR query is transmitted, and has a unit of second.NTPTimestamp which represents a current playback time of content may bedisplayed as a relative time with respect to a start time of the contentand has a unit of millisecond. The representations and units ofQueryInterval and NTPTimestamp are the same in Table 7 and FIG. 9.

Referring to Table 6, the first video display device (device 1) mayrequest a next ACR query at a time point t3a at which 2700 seconds,i.e., 45 minutes, elapse after reception of an ACR query response. Itmay be understood that 900000 milliseconds, i.e., 15 minutes, elapse ata current view time point of the first video display device (device 1)from the start time of the program A.

Referring back to FIG. 34, when a channel is changed at a second timepoint t2 to start playback of the program A, a second video displaydevice (device 2) receives an ACR query response as illustrated in Table7 from the fingerprint server 22.

TABLE 7 <ACR-Result ResultCode= “200” > <ContentID>ProgramA</ContentID><NTPTimestamp>2700000</NTPTimestamp> <QuetyInterval>905</QueryInterval><Confidence>100</Confidence> </ACR-Result>

Referring to Table 7, the second video display device (device 2) mayrequest a next ACR query at a time point t3b at which 905 seconds, i.e.,15 minutes and 5 seconds, elapse after reception of an ACR queryresponse. It may be understood that 2700000 milliseconds, i.e., 45minutes, elapse at a current view time point of the second video displaydevice (device 2) from the start time of the program A.

Here, in order to prevent a plurality of ACR queries from beingconcurrently transmitted from the plurality of video display devices, abroadcaster may differently determines ACR query periods for the videodisplay devices. For example, as shown in Tables 6 and 7, the ACR queryperiods may be set so that there is a time difference of 5 secondsbetween the time at which the first video display device (device 1)transmits a next ACR query and the time at which the second videodisplay device (device 2) transmits a next ACR query.

Referring back to FIG. 34, when the first video display device(device 1) requests a next ACR query at the time point t3a, the firstvideo display device (device 1) receives an ACR query response as shownin Table 8 from the fingerprint server 22. Here, the time point t3a maybe a playback start time point of the program B.

TABLE 8 <ACR-Result ResultCode= “200” > <ContentID>ProgramB</ContentID><NTPTimestamp>0</NTPTimestamp> <QueryInterval>300</QueryInterval><Confidence>100</Confidence> </ACR-Result>

Referring to Table 8, the first video display device (device 1) mayrequest a next ACR query at a fourth time point t4 at which 300 seconds,i.e., 5 minutes, elapse after reception of an ACR query response. It maybe understood that a current view time point of the first video displaydevice (device 1) is a start time point of the program B.

Referring back to FIG. 34, when the second video display device (device2) requests a next ACR query at the time point t3b, the second videodisplay device (device 2) receives an ACR query response as shown inTable 9 from the fingerprint server 22. Here, 5 seconds may elapse atthe time point t3b after start of playback of the program B.

TABLE 9 <ACR-Result ResultCode= “200” > <ContentID>ProgramB</ContentID><NTPTimestamp>5000</NTPTimestamp> <QueryInterval>300</QueryInterval><Confidence>100</Confidence> </ACR-Result>

Referring to Table 9, the second video display device (device 2) mayrequest a next ACR query at a time point (not illustrated) at which 300seconds, i.e., 5 minutes, elapse after reception of an ACR queryresponse. It may be understood that 5000 milliseconds. i.e., 5 minutes,elapse at a current view time point of the first video display device(device 1) from the start time of the program B.

FIG. 35 is a ladder diagram illustrating a data flow for contentrecognition in a fingerprint based network topology according to anotherembodiment.

The content providing server 10 such as a broadcaster/CP extracts afingerprint of content and determines a fingerprint type in operationS2501.

The broadcaster/CP may extract a fingerprint of a program using a toolprovided by an ACR operator, may determine the fingerprint type of theprogram, and then may store the fingerprint type with the extractedfingerprint.

The fingerprint type of the program may be determined as a typeincluding at least one of a video fingerprint and an audio fingerprint.

Thereafter, the content providing server 10 transmits the extractedfingerprint and the determined fingerprint type to the fingerprintserver 22 in operation S2503. Here, the content providing server 10 mapsa content ID for the program to the fingerprint in order to transmit thefingerprint.

The content providing server 10 may transmit the fingerprint and thefingerprint type before the program is broadcasted or at the same timeas the extraction of the fingerprint. For example, the broadcaster/CPmay transmit the fingerprint for a pre-produced program and thefingerprint type to the fingerprint server 22 before the program isbroadcasted. Alternatively, in the case where a live program isbroadcasted, the broadcaster/CP may extract the fingerprint for the liveprogram in real time, and then may transmit the extracted fingerprintand the fingerprint type to the fingerprint server 22.

The fingerprint server 22 stores the received fingerprint and thefingerprint type in operation S2505. The fingerprint server 22 may storethe content ID, the time stamp, the fingerprint and the fingerprint typeof the program transmitted from the broadcaster in an ACR DB.

Thereafter, the video display device 100 extracts a fingerprint from anaudio sample of a partial frame or a partial interval of the main audioand video content, and transmits an ACR query request message includingthe extracted fingerprint to the fingerprint server 22 corresponding toan ACR server in operation S2507.

Here, the fingerprint extracted by the video display device 100 may be adefault-type fingerprint. The default type of the fingerprint extractedby the video display device 100 may be different according to abroadcaster possessing the corresponding content, and the default typesof the fingerprints extracted by the plurality of video display devicesincluded in a network may be different from each other.

The fingerprint server 22 queries the fingerprint included in the ACRquery request message from the ACR DB to identify a programcorresponding to the fingerprint, and provides an ACR query responsemessage including a next fingerprint type for the identified program tothe video display device 100 in operation S2509.

Here, the ACR query response message may include various types ofcontent information in addition to the next fingerprint type.Hereinafter, examples of the content information on the program includedin the ACR query response message will be described with reference toFIG. 36 and Table 10.

FIG. 36 is an XML schema diagram of ACR-Resulttype containing a queryresult according to another embodiment.

As shown in FIG. 36, ACR-Resulttype containing a query result includesResultCode attributes and Content ID, NTPTimestamp, NextFPType andConfidence elements.

For example, if the ResultCode attribute has a value of 200, this maymean that the query result is successful. If the ResultCode attributehas a value of 404, this may mean that the query result is unsuccessful.

The NTPTimestamp element may include at least one of a start time and anend time of a content section from which a fingerprint is extracted.

The NextFPType element represents a next fingerprint type, i.e., a typeof a fingerprint to be extracted by the video display device 100 andmapped in the fingerprint server 22. The NextFPType element mayrepresent a fingerprint type of a corresponding program predetermined bya broadcaster according to a characteristic of content, in operationS2501. Here, the next fingerprint type may be a type including at leastone of a video fingerprint and an audio fingerprint. According to thecharacteristic of content, a single fingerprint type may be determinedfor a single piece of content, or a plurality of different fingerprinttypes may be determined according to a time stamp for a single piece ofcontent.

The Confidence element represents the accuracy of content recognition.The Confidence element may have a value of one of 0 to 100 according toa degree of mapping. For example, when it is confirmed that afingerprint is correctly mapped to a program, the Confidence element mayhave a value of 100.

Table 10 illustrates an XML schema of ACR-ResultType containing thequery result.

TABLE 10 <?xml version=“1.0” encoding=“UTF-8”?> <xs:schemaxmlns:xs=“http://www.w3.org/2001/XMLSchema”elementFormDefault=“qualified” attributeFormDefault=“unqualified”> <xs:complexTypename=“ACR-ResultType”> <xs:sequence> <xs:elementname=“ContentID”type=“xs:anyURI”/> <xs:elementname=“NTPTimestamp”type=“xs:unsignedLong”/> <xs:elementname=“NextFptype”type=“xs:string” minOCCurs=“0”/> <xs:elementname=“Confidence”type=type=“xs:unsignedIat” minOCCurs=“0”/> <xs:anynamespace=“##other” processContents=“skip” minOccurs=“0”maxOccurs=“unbounded”/> </xs:sequence> <xs:attribute name=“ResultCode”type=“xs:string” use=“required”/> <xs:anyAttributeprocessContents=“skip”/> </xs:complexType> </xs:schema>

Hereinafter, FIG. 35 is described again.

Thereafter, the video display device 100 extracts a fingerprint from anaudio sample of a partial frame or a partial interval of the main audioand video content, and transmits an ACR query request message includingthe extracted fingerprint to the fingerprint server 22 corresponding toan ACR server in operation S2511.

Here, the fingerprint extracted by the video display device 100 may be afingerprint of the next fingerprint type included in the ACR queryresponse message received in operation S2509.

The fingerprint server 22 queries the fingerprint included in the ACRquery request message received in operation S2511 from the ACR DB toidentify a program corresponding to the fingerprint, and provides an ACRquery response message including a next fingerprint type for theidentified program to the video display device 100 in operation S2513.

Hereinafter, a criterion for determining a fingerprint type and an ACRquery response according to an embodiment will be described withreference to FIGS. 13 to 15.

Firstly, in the case where similar videos are broadcasted throughvarious channels, such as sports broadcast, it is difficult to identifya program using the video characteristic information. However, since thesimilar videos may have different audio signals due to different sportscommentaries of broadcasters, a program may be identified using theaudio characteristic information. Therefore, each broadcaster maypredetermine the fingerprint type as an audio fingerprint.

In this first case, if a program becomes a subject of viewing due tostart of the program or channel switch, the video display device 100such as a TV transmits an ACR query request message including afingerprint extracted from the program to the fingerprint server 22.Here, the video display device 100 may extract a fingerprint of apredetermined default type or may extract both an audio fingerprint anda vide fingerprint.

The fingerprint server 22 that has received the ACR query requestmessage queries the fingerprint included in the ACR query requestmessage from an ACR DB to identify a program corresponding to thefingerprint, and provides an ACR query response message including apredetermined next fingerprint type of the identified program to thevideo display device 100. Table 11 illustrates the ACR query responseprovided from the fingerprint server 22 to the video display device 100in the first case.

TABLE 11 <ACR-Result ResultCode= “200” > <ContentID>Program1</ContentID><NTPTimestamp>900000</NTPTimestamp> <NextFPType>audio</NextFPType><Confidence>100</Confidence> </ACR-Result>

As shown in Table 11, the next fingerprint type of the first case is anaudio fingerprint type.

Secondly, in the case where a ceremony of a national holiday or a speechis broadcasted or a syndicated program purchased from anotherbroadcaster that has produced and broadcasted the program isbroadcasted, the content itself may be identified through an audiosignal. However, since a broadcaster cannot be identified, the contentaccording to the broadcaster cannot be identified.

For this second case, it may be necessary to identify content and abroadcaster logo using a video fingerprint. Alternatively, the contentmay be identified using an audio fingerprint and the broadcaster logomay be additionally identified using a video fingerprint. However, anembodiment is not limited thereto.

Table 12 illustrates the ACR query response provided from thefingerprint server 22 to the video display device 100 in the secondcase.

TABLE 12 <ACR-Result ResultCode= “200” > <ContentID>Program2</ContentID><NTPTimestamp>900000</NTPTimestamp><NextFPType>audioandvideo</NextFPType> <Confidence>100</Confidence></ACR-Result>

As shown in Table 12, the next fingerprint type of the second case is anaudio and video fingerprint type.

Thirdly, content may not be identified when background music is playedin a soap opera or a movie. In this third cased, if the fingerprint typeis changed according to a time line of the content, the content may bemore clearly identified. Here, if the video display device 100 transmitsan ACR query request according to a predetermined period, thefingerprint server 22 may send an ACR query response including apredetermined fingerprint type at a predetermined time according to atype of the content. Here, the video display device 100 may extract afingerprint of the fingerprint type included in the received ACR queryresponse and may include the extracted fingerprint in a next ACR queryrequest.

Table 13 illustrates the ACR query response provided from thefingerprint server 22 to the video display device 100 in the third case.

TABLE 13 <ACR-Result ResultCode= “200” > <ContentID>Program3</ContentID><NTPTimestamp>300000</NTPTimestamp> <NextFPType>audio</NextFPType><Confidence>100</Confidence> </ACR-Result> <ACR-Result ResultCode=“200” > <ContentID>Program3</ContentID><NTPTimestamp>900000</NTPTimestamp> <NextFPType>video</NextFPType><Confidence>100</Confidence> </ACR-Result>

As shown in Table 13, the next fingerprint type of the third case may bechanged according to a current playback time (NTFTimestamp) of thecontent. For example, when the current playback time of the content is300000 seconds, i.e., 5 minutes, the next fingerprint type may be anaudio fingerprint type, and, when the current playback time of thecontent is 900000 seconds, i.e., 15 minutes, the next fingerprint typemay be a video fingerprint type.

In the case where a video fingerprint is used to identify contentincluding caption or subtitle information, the content identificationmay fail according to a video fingerprint algorithm if the subtitleinformation is overlaid on a video image.

If the content identification is successful even in this case, thismeans noise included content identification. Therefore, a false positivemay occur. The false positive represents the case where different piecesof content are recognized as identical pieces of content. For example,when similar scenes appear between different programs, the programs maybe identified as identical programs.

A fingerprint technique using a sampling window for preventing the errordue to a video fingerprint will be described with reference to FIGS. 37to 40.

FIG. 37 is a ladder diagram illustrating a data flow for contentrecognition in a fingerprint based network topology according to anotherembodiment.

The content providing server 10 such as a broadcaster/CP extracts afingerprint of content and designates a sampling window in operationS2601.

The sampling window represents a region for extracting a videofingerprint from the entire screen of the content. Here, the samplingwindow may be a video region designated away from a space where thesubtitle information is outputted in a program. The broadcaster/CP notonly generates content but also determines a location where the subtitleinformation is outputted on a video image. Therefore, the broadcaster/CPmay notice a region to be sampled for a video fingerprint.

The broadcaster/CP may extract a fingerprint of a program using a toolprovided by an ACR operator, may designate the sampling window accordingto a subtitle information output location of the program, and then maystore the sampling window with the extracted fingerprint.

In the case where the subtitle information is outputted to the samelocation over the entire section of the program, the sampling window maybe always designated the same for the program. On the contrary, in thecase where the subtitle information is outputted to different regionsaccording to a time section of the program, the sampling window may bedifferently designated according to a section of the program.

Thereafter, the content providing server 10 transmits the extractedfingerprint and the designated sampling window to the fingerprint server22 in operation S2603. Here, the content providing server 10 maps acontent ID for the program to the fingerprint in order to transmit thefingerprint.

The content providing server 10 may transmit the fingerprint and thesampling window before the program is broadcasted or at the same time asthe extraction of the fingerprint. For example, the broadcaster/CP maytransmit the fingerprint for a pre-produced program and the designatedsampling window to the fingerprint server 22 before the program isbroadcasted. Alternatively, in the case where a live program isbroadcasted, the broadcaster/CP may extract the fingerprint for the liveprogram in real time, and then may transmit the extracted fingerprintand the sampling window to the fingerprint server 22.

The fingerprint server 22 stores the received fingerprint and thedesignated sampling window in operation S2605. The fingerprint server 22may store the content ID, the time stamp, the fingerprint and thesampling window of the program transmitted from the broadcaster in anACR DB.

Thereafter, the video display device 100 extracts a fingerprint from anaudio sample of a partial frame or a partial interval of the main audioand video content, and transmits an ACR query request message includingthe extracted fingerprint to the fingerprint server 22 corresponding toan ACR server in operation S2607.

The fingerprint server 22 queries the fingerprint included in the ACRquery request message from the ACR DB to identify a programcorresponding to the fingerprint, and provides an ACR query responsemessage including a first sampling window for the identified program tothe video display device 100 in operation S2609.

Here, the ACR query response message may include various types ofcontent information in addition to the sampling window. Hereinafter,examples of the content information on the program included in the ACRquery response message will be described with reference to FIG. 38 andTable 14.

FIG. 38 is an XML schema diagram of ACR-Resulttype containing a queryresult according to another embodiment.

As shown in FIG. 38, ACR-Resulttype containing a query result includesResultCode attributes and Content ID, NTPTimestamp, SamplingWindow andConfidence elements.

For example, if the ResultCode attribute has a value of 200, this maymean that the query result is successful. If the ResultCode attributehas a value of 404, this may mean that the query result is unsuccessful.

The NTPTimestamp element may include at least one of a start time and anend time of a content section from which a fingerprint is extracted.

The SamplingWindow element represents a valid time, location, and sizeof a sampling window required for an ACR query.

The SamplingWindow element may include an x element, a y element, awidth element and a height element, wherein a unit of each element is apixel. The x element may be an x coordinate of an upper left side of thesampling window. The y element may be a y coordinate of the upper leftside of the sampling window. The width element may be a width of thesampling window. The height element may be a height of the samplingwindow.

Furthermore, the SamplingWindow element of FIG. 38 may include aValidFrom element and a ValidTo element. The ValidFrom element may be avalid sampling start time point of the sampling window. The ValidToelement may be a valid sampling end time point of the sampling window.The video display device 100 may use the sampling window correspondingto a valid time of the sampling window, i.e., a time between the validsampling start time of the sampling window and the valid sampling endtime of the sampling window.

The Confidence element represents the accuracy of content recognition.The Confidence element may have a value of one of 0 to 100 according toa degree of mapping. For example, when it is confirmed that afingerprint is correctly mapped to a program, the Confidence element mayhave a value of 100.

Table 14 illustrates an XML schema of ACR-ResultType containing thequery result.

TABLE 14 <?xml version=“1.0” encoding=“UTF-8”?> <xs:schemaxmlns:xs=“http://www.w3.org/2001/XMLSchema”elementFormDefault=“qualified” attributeFormDefault=“unqualified”><xs:complexType name=“ACR-ResultType1”> <xs:sequence> <xs:elementname=“ContentID”type=“xs:anyURI”/> <xs:elementname=“NTPTimestamp”type=“xs:unsignedLong”/> <xs:elementname=“SamplingWindow”type=“SamplingWindowType” minOccurs=“0”maxOccurs=“unbounded”/> <xs:elementname=“Confidence”type=“xs:unsignedInt” minOccurs=“0”/> <xs:anynamespace=“##other” processContents=“skip” minOccurs=“0”maxOccurs=“unbounded”/> </xs:sequence> <xs:attributename=“ResultCode”type=“xs:string” use=“required”/> <xs:anyAttributeprocessContents=skip”/> </xs:complexType> </xs:complexTypename=“SamplingWindowType”> <xs:sequence> <xs:element name=“x”type=“xs:unsignedLong”/> <xs:element name=“y” type=“xs:unsignedLong”/><xs:element name=“width” type=“xs:unsignedLong”/> <xs:elementname=“height” type=“xs:unsignedLong”/> </xs:sequence> <xs:attributename=“ValidFrom” type=“xs:unsignedLong” use=“required”/> <xs:attributename=“ValidTo” type=“ type=“xs:unsignedLong” use=“required”/></xs:complexType> </xs:schema>

Hereinafter, FIG. 37 is described again.

Thereafter, the video display device 100 extracts a fingerprint from anaudio sample of a partial frame or a partial interval of the main audioand video content, and transmits an ACR query request message includingthe extracted fingerprint to the fingerprint server 22 corresponding toan ACR server in operation S2611.

Here, the fingerprint extracted by the video display device 100 may be afingerprint by the sampling window included in the ACR query responsemessage received in operation S2609.

The fingerprint server 22 queries the fingerprint included in the ACRquery request message received in operation S2611 from the ACR DB toidentify a program corresponding to the fingerprint, and provides an ACRquery response message including a second sampling window for theidentified program to the video display device 100 in operation S2613.

Here, the SamplingWindow elements of the first sampling window and thesecond sampling window may be the same or different from each otheraccording to whether programs are the same, whether a program has thesubtitle information, and whether a location of the subtitle informationis changed.

In the case where a video fingerprint is used to identify content, thecontent identification may fail according to a video fingerprintalgorithm, if an on screen display (OSD) is overlaid on a program videotransmitted to the video display device 100. The on screen display mayinclude a menu, an electronic program guide (EPG), a channel bar and awidget. The on screen display may represent information provided from anoperator excepting the content providing server 10 and the fingerprintserver 22. Hereinafter, the operator may represent an operator exceptingthe content providing server 10 and the fingerprint server 22, such asthe MVPD operator 30 or the broadcast receiving device 60. Due to the onscreen display, the above-mentioned false positive may occur.

Hereinafter, a method, in which the MVPD 30 connected to the videodisplay device 100 is estimated and a sampling window is designatedaccording to GUI characteristic information on the estimated MVPD 30 inorder to prevent the occurrence of the false positive, will be describedwith reference to FIGS. 39 and 40.

FIG. 39 is a ladder diagram illustrating a data flow for contentrecognition in a fingerprint based network topology according to anotherembodiment.

The content providing server 10 such as a broadcaster/CP extracts afingerprint of content in operation S2701.

The broadcaster/CP may extract a fingerprint of a program using a toolprovided by an ACR operator, and may store the extracted fingerprintwith a content ID and a time stamp of the program.

Thereafter, the content providing server 10 transmits the extractedfingerprint to the fingerprint server 22 in operation S2703. Here, thecontent providing server 10 maps a content ID for the program to thefingerprint in order to transmit the fingerprint.

The content providing server 10 may transmit the fingerprint before theprogram is broadcasted or at the same time as the extraction of thefingerprint. For example, the broadcaster/CP may transmit thefingerprint for a pre-produced program to the fingerprint server 22before the program is broadcasted. Alternatively, in the case where alive program is broadcasted, the broadcaster/CP may extract thefingerprint for the live program in real time, and then may transmit theextracted fingerprint to the fingerprint server 22.

The fingerprint server 22 stores the received fingerprint in operationS2705. The fingerprint server 22 may store the content ID, the timestamp and the fingerprint of the program transmitted from thebroadcaster in an ACR DB.

Furthermore, the fingerprint server 22 obtains GUI characteristicinformation for each operator from the content providing server 10 inoperation S2707. The GUI characteristic information representsinformation on a location, a size and a shape with respect to an onscreen display. The GUI characteristic information may be provided froma broadcaster, may be transferred from a third party operator, or may bedirectly detected by an ACR operator.

Thereafter, the video display device 100 extracts a fingerprint from anaudio sample of a partial frame or a partial interval of the main audioand video content, and transmits an ACR query request message includingthe extracted fingerprint to the fingerprint server 22 corresponding toan ACR server in operation S2709.

The fingerprint server 22 queries the fingerprint included in the ACRquery request message from an ACR DB to identify a program correspondingto the fingerprint in operation S2711.

When there is no program matched to the fingerprint, the fingerprintserver 22 analyzes a list of failed fingerprints among fingerprintstransmitted by the video display device 100, so as to estimate anoperator that has provided the on screen display in operation S2713. Thefingerprint server 22 compares pieces of the GUI characteristicinformation for each operator by analyzing the list of failedfingerprints, and, as a result of the comparison, estimates the operatorthat has provided the on screen display.

The fingerprint server 22 retrieves the GUI characteristic informationof the estimated operator, and generates a sampling window according tothe retrieved GUI characteristic information in operation S2715.

Thereafter, the fingerprint server 22 transmits an ACR query requestmessage including recognition failure and the generated sampling windowto the video display device 100 in operation S2717.

Here, the ACR query response message may include various types ofcontent information in addition to the recognition failure and thesampling window. Hereinafter, examples of the content information on theprogram included in the ACR query response message will be describedwith reference to FIG. 40 and Table 15.

FIG. 40 is an XML schema diagram of ACR-Resulttype containing a queryresult according to another embodiment.

As shown in FIG. 40, ACR-Resulttype containing a query result includesResultCode attributes and Content ID, NTPTimestamp, SamplingWindow andConfidence elements.

For example, if the ResultCode attribute has a value of 200, this maymean that the query result is successful. If the ResultCode attributehas a value of 404, this may mean that the query result is unsuccessful.The ResultCode attribute included in the ACR query response messagetransmitted in operation S2717 of FIG. 39 may have a value of 404.

The NTPTimestamp element may include at least one of a start time and anend time of a content section from which a fingerprint is extracted.

The SamplingWindow element represents a location and a size of asampling window required for an ACR query.

The SamplingWindow element may include an x element, a y element, awidth element and a height element, wherein a unit of each element is apixel. The x element may be an x coordinate of an upper left side of thesampling window. The y element may be a y coordinate of the upper leftside of the sampling window. The width element may be a width of thesampling window. The height element may be a height of the samplingwindow.

Furthermore, the SamplingWindow element of FIG. 40 may include a guiTypeelement. The guiType element may display a GUI type of an on screendisplay, such as a menu and an electronic program guide, as characterstrings. Here, the corresponding content may have a plurality ofdifferent sampling windows according to the guiType element.

The Confidence element represents the accuracy of content recognition.The Confidence element may have a value of one of 0 to 100 according toa degree of mapping. For example, when it is confirmed that afingerprint is correctly mapped to a program, the Confidence element mayhave a value of 100.

Table 15 illustrates an XML schema of ACR-ResultType containing thequery result.

TABLE 15 <?xml version=“1.0” encoding=“UTF-8”?> <xs:schemaxmlns:xs=“http://www.w3.org/2001/XMLSchema”elementFormDefault=“qualified” attributeFormDefault=“unqualified”><xs:complexType name=“ACR-ResultType2”> <xs:sequence> <xs:elementname=“ContentID”type=“xs:anyURI”/> <xs:elementname=“NTPTimestamp”type=“xs:unsignedLong”/> <xs:elementname=“SamplingWindow”type=“SamplingWindowType2” minOccurs=“0”maxOccurs=“unbounded”/> <xs:elementname=“Confidence”type=“xs:unsignedInt” minOccurs=“0”/> <xs:anynamespace=“##other” processContents=“skip” minOccurs=“0”maxOccurs=“unbounded”/> </xs:sequence> <xs:attributename=“ResultCode”type=“xs:string” use=“required”/> <xs:anyAttributeprocessContents=skip”/> </xs:complexType> </xs:complexTypename=“SamplingWindowType2”> <xs:sequence> <xs:element name=“x”type=“xs:unsignedLong”/> <xs:element name=“y” type=“xs:unsignedLong”/><xs:element name=“width” type=“xs:unsignedLong”/> <xs:elementname=“height” type=“xs:unsignedLong”/> </xs:sequence> <xs:attributename=“guiType” type=“xs:string” use=“optional”> </xs:complexType></xs:schema>

Hereinafter, FIG. 39 is described again.

Thereafter, the video display device 100 extracts a fingerprint from anaudio sample of a partial frame or a partial interval of the main audioand video content, and transmits an ACR query request message includingthe extracted fingerprint to the fingerprint server 22 corresponding toan ACR server in operation S2719.

Here, the fingerprint extracted by the video display device 100 may be afingerprint by the sampling window included in the ACR query responsemessage received in operation S2717.

The fingerprint server 22 queries the fingerprint included in the ACRquery request message received in operation S2719 from the ACR DB toidentify a program corresponding to the fingerprint, and provides an ACRquery response message including recognition success to the videodisplay device 100 in operation S2721.

Pieces of content may have different characteristics. Accordingly,pieces of content may have different characteristic information typesfor efficient recognition of content. For example, in the case of newscontent, broadcasters may broadcast different audios based on the samevideo material. In the case of sports content, the broadcasters maybroadcast different sports commentaries based on the same video materialTherefore, in the case of the news content and the sports content, it isefficient for a video display device to recognize content based on audiocharacteristic information. However, in the case of movie content, aplurality of audio tracks may be included. Therefore, even in the samemovie, the audio characteristic information may be changed according toselection of an audio track. Therefore, in the case of the moviecontent, it is efficient for the video display device to recognizecontent based on video characteristic information. Therefore, it may benecessary to develop a video display device capable of determiningcharacteristic information for efficiently recognizing content. Inparticular, such an issue may arise when genres of content are differentfrom each other. Therefore, it may be necessary to determine thecharacteristic information type based on the genre of content.

A video display device for determining characteristic information forefficiently recognizing content will be described with reference toFIGS. 41 to 46.

FIG. 41 is a diagram illustrating a network topology in which the videodisplay device determines a characteristic information type according toan embodiment.

The network topology includes the video display device 100, the contentrecognizing service providing server 20, and the third party server 90.

The video display device 100 extracts characteristic information frombroadcast content including at least one of an uncompressed video and anuncompressed audio obtained from an external input. Here, the videodisplay device 100 may extract the characteristic information from thebroadcast content including at least one of the uncompressed video andthe uncompressed audio based on a first characteristic information type.The video display device 100 transmits a first ACR query request messageto the content recognizing service providing server 20 based on thecharacteristic information. The video display device 100 receives afirst ACR query response message for the first ACR query request messagefrom the content recognizing service providing server 20. Here, thefirst ACR query response message includes content information. Thecontent information may include at least one of a URL of a leanbackapplication for the content, a content identifier, a content timestamp,and a content recognition accuracy. Here, the content information mayhave the same format as described above with reference to FIG. 33 or 38.The video display device 100 may determine a second characteristicinformation type based on the first ACR query response message. Indetail, the video display device 100 may extract the content identifierfrom the first response message. The video display device 100 may obtaina genre of content based on the content identifier. In detail, the videodisplay device 100 transmits a query request message based on thecontent identifier to the third party server 90. The video displaydevice 100 receives a query response message based on a recommendedcharacteristic information type from the third party server 90. Thevideo display device 100 determines the second characteristicinformation type based on the recommended characteristic informationtype. The video display device 100 may extract the characteristicinformation from the broadcast content including at least one of theuncompressed video and the uncompressed audio based on the secondcharacteristic information type. The video display device 100 transmitsa second ACR query request message to the content recognizing serviceproviding server 20 based on the second characteristic information type.In detail, the video display device 100 transmits the second ACR queryrequest message including characteristic information corresponding tothe second characteristic information type to the content recognizingservice providing server 20. The video display device 100 receives asecond ACR query response message from the content recognizing serviceproviding server 20.

The content recognizing service providing server 20 stores thecharacteristic information of content. The content recognizing serviceproviding server 20 receives the ACR query request message from thevideo display device 100. The content recognizing service providingserver 20 transmits the ACR query response message based on the contentcharacteristic information. In detail, the content recognizing serviceproviding server 20 transmits the ACR query response message includingthe content information to the video display device 100.

The third party server 90 receives the query request message based onthe content identifier to the video display device 100. In detail, thethird party server may receives the query request message including thecontent identifier. The third party server 90 transmits the queryresponse message to the video display device 100 based on therecommended characteristic information type. The third party server 90may transmit the query response message including the recommendedcharacteristic information type to the video display device 100.

Operations of the video display device 100, the content recognizingservice providing server 20, and the third party server 90 will bedescribed in more detail with reference to FIG. 43.

FIG. 42 is a ladder diagram illustrating that the video display devicedetermines a characteristic information type according to an embodiment.

The video display device 100 is turned on to start to operate inoperation S2801. The video display device 100 may obtain uncompressedaudio and video content through an external input terminal. Here, theuncompressed audio and video content may include at least one of anuncompressed audio and an uncompressed video. The video display device100 may display the uncompressed audio and video content through thedisplay unit 121.

The video display device 100 transmits the first ACR query requestmessage to the content recognizing service providing server 20 based onthe first characteristic information type in operation S2803. In detail,the video display device 100 may extract characteristic informationcorresponding to the first characteristic information type from theuncompressed audio and video content input through an external input,and may transmit the ACR query request message including thecharacteristic information to the content recognizing service providingserver 20. Here, the first characteristic information type may be adefault characteristic information type.

The content recognizing service providing server 20 transmits the firstACR query response message to the video display device 100 in operationS2805. In detail, the content recognizing service providing server 20may transmit the first ACR query response message including the contentinformation to the video display device 100. Here, the contentinformation may include at least one of a URL of a leanback applicationfor the content, a content identifier, a content timestamp, and acontent recognition accuracy. Here, the content information may have thesame format as described above with reference to FIG. 33 or 38.

The video display device 100 transmits the query request message to thethird party server 90 based on the content identifier in operationS2807. In detail, the video display device 100 may transmit the queryrequest message including the content identifier to the third partyserver 90.

The video display device 100 obtains at least one of the genre ofcontent and the second characteristic information type based on thecontent identifier.

The third party server 90 transmits the query response message to thevideo display device 100 based on the recommended characteristicinformation type of content in operation S2809. In detail, the thirdparty server 90 may transmit the query response message including therecommended characteristic information type to the video display device100. In a specific embodiment, the third party server 90 may includemetadata of content. The third party server 90 may extracts the genre ofcontent from the metadata of content corresponding to the contentidentifier.

Here, the content identifier may be TMS ID. A TMS is a service providerfor providing metadata for content. A number of broadcasters andbroadcast receiver companies provide enhanced services by usingbroadcast data provided by the TMS. In particular, the broadcasters andbroadcast receiver companies provide an EPG service by using the TMS ID.The TMS ID is an identifier used by the TMS to identify content. If theTMS IDs of a plurality of pieces of content are the same, a broadcastreceiver or a video display device may determine that the plurality ofpieces of content are the same. In detail, the third party server 90 mayobtain the genre of content from TMS data provided by the TMS based onthe TMS ID. In another specific embodiment, the content identifier maybe a content reference identifier (CRID). The CRID which is a contentidentifier defined by TV-Anytime is used by broadcasters and broadcastreceiver companies of some countries to provide an EPG service and ascheduled recording service. In detail, the third party server 90 mayobtain the genre of content from EPG data based on the CRID. Here, theEPG data may be EIT. In detail, the EPG data may be a genre descriptordefined by an ATSC standard. Alternatively, the EPG data may be acontent descriptor defined by a DVB standard.

According to a specific embodiment, the third party server 90 maydetermine the recommended characteristic information type that serves asa basis for determining the second characteristic information based onthe genre of content. In detail, the third party server 90 may determinethe recommended characteristic information type with reference to thefollowing table.

TABLE 16 Genre FP Type News Audio FP Sports . . . Music Video FPDocumentary . . . Drama Audio FP + Video FP . . .

For example, in the case where the genre of content is news or sports,the third party server 90 may determine the recommended characteristicinformation type as audio characteristic information. For example, inthe case where the genre of content is music or documentary, the thirdparty server 90 may determine the recommended characteristic informationtype as video characteristic information. For example, in the case wherethe genre of content is a soap opera, the third party server 90 maydetermine that the recommended characteristic information type includesboth the audio characteristic information and the video characteristicinformation. The third party server 90 may transmit the query responsemessage based on the recommended characteristic information type. Thevideo display device 100 may efficiently recognize each content throughthe recommended characteristic information type determined as describedabove. In another specific embodiment, the third party server 90 maytransmit the query response message to the video display device 100based on the genre of content, without determining the characteristicinformation type. This operation will be described in more detail withreference to FIGS. 44 and 45.

The video display device 100 determines the second characteristicinformation type based on at least one of recommended characteristicinformation, a constraint on extraction of characteristic information,and an accuracy of content recognition in operation S2811. The contentrecognizing service providing server 20 and a matching algorithm may beused in common, but hardware of the video display device 100 may bedifferent depending on a manufacturer. Therefore, the video displaydevice 100 may have a hardware constraint for the audio characteristicinformation and the video characteristic information during amanufacturing process. Furthermore, when a function that is not acharacteristic information extracting function uses an audio driver or avideo driver while the video display device 100 operates, the videodisplay device 100 may be temporarily unable to extract the audiocharacteristic information or the video characteristic information.Therefore, in order to overcome this limitation, the video displaydevice 100 may determine the second characteristic information typebased on the constraint on extraction of the characteristic informationof the video display device 100. In a specific embodiment, in the casewhere the characteristic information corresponding to the recommendedcharacteristic information type is unable to be extracted, the videodisplay device 100 may determine the second characteristic informationtype as a characteristic information type that enables extraction of thecharacteristic information. For example, in the case where the videocharacteristic information is unable to be extracted since all videodrivers of the video display device 100 are currently used, and therecommended characteristic information type is video characteristicinformation, the video display device 100 may determine the secondcharacteristic information type as audio characteristic information.

Although the second characteristic information type is the audiocharacteristic information, the video display device 100 may fail torecognize content based on the audio characteristic information in thecase where a mute interval is included in the content or an intervalthat is represented by general voice information alone such asbackground music is long. Furthermore, although the secondcharacteristic information type is designated as the videocharacteristic information, the video display device 100 may fail torecognize content in the case where there are few differences betweenframes of the content such as a still image. In order to overcome thislimitation, in another specific embodiment, the video display device 100may determine the second characteristic information type based on theaccuracy of content recognition. This will be described in more detailwith reference to FIG. 43. FIG. 43 is a flowchart illustrating that thevideo display device determines the characteristic information typebased on the accuracy of content recognition according to an embodiment.

The video display device 100 extracts the accuracy of contentrecognition based on the first ACR query response message in operationS2851. In a specific embodiment, the first ACR query response messagemay include the accuracy of content recognition. In detail, the accuracyof content recognition may be included in the form of the confidenceelement as described above.

The video display device 100 determines whether the accuracy of contentrecognition is smaller than a preset recognition reference value inoperation S2853. Here, the preset recognition reference value may bechanged based on a user input.

If the accuracy of content recognition is smaller than the presetrecognition reference value, the video display device 100 determines thesecond characteristic information type as a type different from thefirst characteristic information type in operation S2855. For example,in the case where the first characteristic information type is videocharacteristic information, the accuracy of content recognition is about60%, and the preset recognition reference value is about 70%, the videodisplay device 100 may determine the second characteristic informationtype as an audio characteristic information type.

The operation of determining the characteristic information type basedon at least one of the constraint on extraction of characteristicinformation and the accuracy of content recognition may be repeatedlyperformed during a process of transmitting the ACR query request messageand receiving the ACR query response message after transmitting thesecond ACR query request message.

The video display device 100 transmits the second ACR query requestmessage based on the second characteristic information type in operationS2813. In detail, the video display device 100 may extractcharacteristic information corresponding to the second characteristicinformation type from the uncompressed audio and video content inputthrough an external input, and may transmit the ACR query requestmessage including the characteristic information to the contentrecognizing service providing server 20.

The content recognizing service providing server 20 transmits the secondACR query response message in operation S2815. In detail, the contentrecognizing service providing server 20 may transmit the second ACRquery response message including the content information to the videodisplay device 100. Here, the content information may include at leastone of the URL of the leanback application for the content, the contentidentifier, the content timestamp, and the content recognition accuracy.

Here, the content recognizing service providing server 20 may transmitthe second characteristic information type and the content informationtogether. In particular, in the case where the second characteristicinformation type is not supported by the content recognizing serviceproviding server 20, the content recognizing service providing server 20may transmit the recommended characteristic information type ascharacteristic information supported by the content recognizing serviceproviding server 20. Thereafter, the content recognizing serviceproviding server 20 may transmit the recommended characteristicinformation type and the content information together. For example, inthe case where the content recognizing service providing server 20 doesnot support the audio characteristic information even though the secondcharacteristic information type is the audio characteristic information,the content recognizing service providing server 20 may transmit thevideo characteristic information as the recommended characteristicinformation.

It will be described that the third party server 90 transmits the queryresponse message to the video display device 100 based on the genre ofcontent, without determining the characteristic information type, withreference to FIGS. 44 and 45.

FIG. 44 is a diagram illustrating a network topology in which the videodisplay device determines a characteristic information type according toanother embodiment. The embodiment of FIG. 44 is only different from theembodiment of FIG. 41 in that the video display device 100 does notobtain the recommended characteristic information from the third partyserver 90 but receives the query response message based on the genre ofcontent from the third party server 90. The same configurations andoperations are thus omitted below.

The video display device 100 receives the content genre from the thirdparty server 90. The video display device 100 may determine the secondcharacteristic information type based on at least one of the contentgenre, the content recognition accuracy, and the constraint onextraction of characteristic information. The video display device 100may determine the second characteristic information type according tothe content genre in the same manner as the third parity server 90 ofFIGS. 41 and 42 determines the recommended characteristic informationtype. Furthermore, in the same manner as described above with referenceto FIGS. 41 and 42, the video display device 100 may determine thesecond characteristic information type based on the content recognitionaccuracy. In addition, the video display device 100 may determine thesecond characteristic information type based on the constraint onextraction of characteristic information of the video display device.

The third party server 90 transmits the first ACR query response messagebased on the content genre. In detail, the third party server 90 maytransmit the first ACR query response message including the contentgenre.

FIG. 45 is a ladder diagram illustrating that the video display devicedetermines the characteristic information type according to anotherembodiment. The embodiment of FIG. 45 is only different from theembodiment of FIG. 42 in that the video display device 100 does notobtain the recommended characteristic information from the third partyserver 90 but receives the query response message based on the genre ofcontent from the third party server 90. The same configurations andoperations are thus omitted below.

The third party server 90 transmits the query response message based onthe content genre. In detail, the third party server 90 may transmit thequery response message including the content genre.

The video display device 100 determines the second characteristicinformation type based on at least one of the content genre, the contentrecognition accuracy, and the constraint on extraction of characteristicinformation in operation S2911. The video display device 100 maydetermine the second characteristic information type according to thecontent genre in the same manner as the third parity server 90 of FIGS.41 and 42 determines the recommended characteristic information type.Furthermore, in the same manner as described above with reference toFIGS. 41 and 42, the video display device 100 may determine the secondcharacteristic information type based on the content recognitionaccuracy. In addition, the video display device 100 may determine thesecond characteristic information type based on the constraint onextraction of characteristic information of the video display device.

FIG. 46 is a conceptual diagram illustrating a video display deviceaccording to an embodiment.

As illustrated in FIG. 46, the video display device 100 according to anembodiment includes an audio/video driver 601, an ACR engine 603, ACRmiddleware 605, an ACR user interface 607, and an ACR plugin 609.

The audio/video driver 601 captures an audio/video signal input as anexternal input of a TV and transfers the audio/video signal to the ACRengine 603. The audio/video driver 601 may provide API so that the ACRengine 603 accesses an audio/video buffer. The audio/video driver 601may also provide characteristic information of audio/video received asan external input such as HDMI. For example, in the case of audio, theaudio/video driver 601 may provide information such as a sample depth, asampling rate, the number of channels (mono/stereo etc.), the number ofsamples, and a sample time. In the case of video, the audio/video driver601 may provide information such as a video format and a width, height,stride, stream time, input type (HDMI1, HDMI2, composite, component,etc.), and the number of frames per second of input video.

The ACR engine 603 may be executed as a separate process, and recognizesa program using the above-mentioned various methods. The ACR engine 603may extract a signature from an audio/video sample, and may transfer theextracted signature to an ACR server so as to recognize content. Asdescribed above, the ACR engine 603 may call API capable of accessingthe audio/video buffer according to platform implementation, and maytransmit an audio/video stream captured by the audio/video driver 601 toan ACR process by using a socket. The ACR engine 603 sends an ACR queryrequest to the ACR server with the signature, and receives, from the ACRsever, an ACR query response including information that indicateswhether content recognition is successful, a content ID, a globalcontent ID, a timestamp, a leanback application URL, a broadcasterserver URL, a promotional application URL, and a content reminderapplication URL. The ACR engine 603 may transfer a result received fromthe ACR server to the ACR middleware 605.

The ACR middleware 605, which is a module corresponding to middleware ofa DTV process, processes ACR control and processes the ACR queryresponse from the ACR engine 603. The ACR middleware 605 controls theACR by executing or terminating the ACR process or by starting orstopping the ACR engine 603. Furthermore, by parsing the ACR queryresponse, the ACR middleware 605 stores values of the content ID, globalcontent ID, and timestamp. The ACR middleware 605 may provide API so asto send the stored values to the ACR UI, or may send the stored valuesto the ACR user interface 607 by means of a message queue and globalparameters. Furthermore, in order to transfer data such as the contentID, the global content ID, and the timestamp to an ACR application suchas a Live+ application or a leanback application, the ACR middleware 605may transfer the data to a web browser through inter-processcommunication (IPC) of a shared memory and a socket.

The ACR user interface (UI) 607 may represent UI for controlling ACRsuch as ACR on/off to a user, or may represent, to the user, informationsuch as a name and time of a recognized program in a status bar UI of aTV.

When the ACR middleware 605 transfers, to a browser, the data such asthe content ID, the global content ID, and the timestamp to betransferred to an ACR application, these values are received by the ACRplugin 609. The ACR plugin 609 may transfer the values to the ACRapplication through a plugin object interface.

According to an embodiment, the third party server 90 may includemetadata of content as described above. The third party server 90 mayinclude ACR configuration information. The third party server 90 maytransmit at least one of the content genre and the recommendedcharacteristic information type to the video display device 100. Thisoperation has been described above in detail, and is thus omitted here.

Next, a structure of a video display device according to variousembodiments will be described with reference to FIGS. 47 and 48.

FIG. 47 is a block diagram illustrating a structure of a fingerprintbased video display device according to another embodiment.

As shown in FIG. 47 a tuner 501 extracts a symbol from an 8-VSB RFsignal transmitted through an air channel.

An 8-VSB demodulator 503 demodulates the 8-VSB symbol that the tuner 501extracts and restores meaningful digital data.

A VSB decoder 505 decodes the digital data that the 8-VSB demodulator503 to restore an ATSC main service and ATSC M/H service.

An MPEG-2 TI Demux 507 filters a Transport Packet that the video displaydevice 100 is to process from an MPEG-2 Transport Packet transmittedthrough an 8-VSB signal or an MPEG-2 Transport Packet stored in a PVRStorage to relay the filtered Transport Packet into a processing module.

A PES decoder 539 buffers and restores a Packetized Elementary Streamtransmitted through an MPEG-2 Transport Stream.

A PSI/PSIP decoder 541 buffers and analyzes PSI/PSIP Section Datatransmitted through an MPEG-2 Transport Stream. The analyzed PSI/PSIPdata are collected by a Service Manager (not shown), and then, is storedin DB in a form of Service Map and Guide data.

A DSMCC Section Buffer/Handler 511 buffers and processes DSMCC SectionData for file transmission through MPEG-2 TP and IP Datagramencapsulation.

An IP/UDP Datagram Buffer/Header Parser 513 buffers and restores IPDatagram, which is encapsulated through DSMCC Addressable section andtransmitted through MPEG-2 TP to analyze the Header of each Datagram.Additionally, an IP/UDP Datagram Buffer/Header Parser 513 buffers andrestores UDP Datagram transmitted through IP Datagram, and then analyzesand processes the restored UDP Header.

A Stream component handler 557 may include ES Buffer/Handler, PCRHandler, STC module, Descrambler, CA Stream Buffer/Handler, and ServiceSignaling Section Buffer/Handler.

The ES Buffer/Handler buffers and restores an Elementary Stream such asVideo and Audio data transmitted in a PES form to deliver it to a properA/V Decoder.

The PCR Handler processes Program Clock Reference (PCR) Data used forTime synchronization of Audio and Video Stream.

The STC module corrects Clock values of the A/V decoders by using aReference Clock value received through PCR Handler to perform TimeSynchronization.

When scrambling is applied to the received IP Datagram, the Descramblerrestores data of Payload by using Encryption key delivered from the CAStream Handler.

The CA Stream Buffer/Handler buffers and processes Data such as Keyvalues for Descrambling of EMM and ECM, which are transmitted for aConditional Access function through MPEG-2 TS or IP Stream. An output ofthe CA Stream Buffer/Handler is delivered to the Descrambler, and then,the descrambler descrambles MPEG-2 TP or IP Datagram, which carriers A/VData and File Data.

The Service Signaling Section Buffer/Handler buffers, restores, andanalyzes NRT Service Signaling Channel Section Data transmitted in aform of IP Datagram. The Service Manager (not shown) collects theanalyzed NRT Service Signaling Channel Section data and stores them inDB in a form of Service Map and Guide data.

The A/V Decoder 561 decodes the Audio/Video data received through an ESHandler to present them to a user.

An MPEG-2 Service Demux (not shown) may include an MPEG-2 TPBuffer/Parser, a Descrambler, and a PVR Storage module.

An MPEG-2 TP Buffer/Parser (not shown) buffers and restores an MPEG-2Transport Packet transmitted through an 8-VSB signal, and also detectsand processes a Transport Packet Header.

The Descrambler restores the data of Payload by using an Encryption key,which is delivered from the CA Stream Handler, on the Scramble appliedPacket payload in the MPEG-2 TP.

The PVR Storage module stores an MPEG-2 TP received through an 8-VSBsignal at the user's request and outputs an MPEG-2 TP at the user'srequest. The PVR storage module may be controlled by the PVR manager(not shown).

The File Handler 551 may include an ALC/LCT Buffer/Parser, an FDTHandler, an XML Parser, a File Reconstruction Buffer, a Decompressor, aFile Decoder, and a File Storage.

The ALC/LCT Buffer/Parser buffers and restores ALC/LCT data transmittedthrough a UDP/IP Stream, and analyzes a Header and Header extension ofALC/LCT. The ALC/LCT Buffer/Parser may be controlled by an NRT ServiceManager (not shown).

The FDT Handler analyzes and processes a File Description Table of FLUTEprotocol transmitted through an ALC/LCT session. The FDT Handler may becontrolled by an NRT Service Manager (not shown).

The XML Parser analyzes an XML Document transmitted through an ALC/LCTsession, and then, delivers the analyzed data to a proper module such asan FDT Handler and an SG Handler.

The File Reconstruction Buffer restores a file transmitted through anALC/LCT, FLUTE session.

If a file transmitted through an ALC/LCT and FLUTE session iscompressed, the Decompressor performs a process to decompress the file.

The File Decoder decodes a file restored in the File ReconstructionBuffer, a file decompressed in the decompressor, or a film extractedfrom the File Storage.

The File Storage stores or extracts a restored file if necessary.

The M/W Engine (not shown) processes data such as a file, which is notan A/V Stream transmitted through DSMCC Section and IP Datagram. The M/WEngine delivers the processed data to a Presentation Manager module.

The SG Handler (not shown) collects and analyzes Service Guide datatransmitted in an XML Document form, and then, delivers them to the EPGManager.

The Service Manager (not shown) collects and analyzes PSI/PSIP Datatransmitted through an MPEG-2 Transport Stream and Service SignalingSection Data transmitted through an IP Stream, so as to produce aService Map. The Service Manager (not shown) stores the produced servicemap in a Service Map & Guide Database, and controls an access to aService that a user wants. The Service Manager is controlled by theOperation Controller (not shown), and controls the Tuner 501, the MPEG-2TP Demux 507, and the IP Datagram Buffer/Handler 513.

The NRT Service Manager (not shown) performs an overall management onthe NRT service transmitted in an object/file form through a FLUTEsession. The NRT Service Manager (not shown) may control the FDT Handlerand File Storage.

The Application Manager (not shown) performs overall management onApplication data transmitted in a form of object and file.

The UI Manager (not shown) delivers a user input to an OperationController through a User Interface, and starts a process for a servicethat a user requests.

The Operation Controller (not shown) processes a command of a user,which is received through a UI Manager, and allows a Manager of anecessary module to perform a corresponding action.

The Fingerprint Extractor 565 extracts fingerprint characteristicinformation from an AV stream.

The Fingerprint Comparator 567 compares the characteristic informationextracted by the Fingerprint Extractor with a Reference fingerprint tofind an identical content. The Fingerprint Comparator 567 may use aReference fingerprint DB stored in local and may query a Fingerprintquery server on the internet to receive a result. The matched resultdata obtained by a comparison result may be delivered to Application andused.

As an ACR function managing module or an application module providing anenhanced service on the basis of ACR, the Application 569 identifies abroadcast content in watching to provide an enhanced service related toit.

FIG. 48 is a block diagram illustrating a structure of a watermark basedvideo display device according to another embodiment.

Although the watermark based video display device of FIG. 48 is similarto the fingerprint based video display device of FIG. 48, thefingerprint based video display device does not includes the FingerprintExtractor 565 and the Fingerprint Comparator 567, but further includesthe Watermark Extractor 566.

The Watermark Extractor 566 extracts data inserted in a watermark formfrom an Audio/Video stream. The extracted data may be delivered to anApplication and may be used.

According to an embodiment, enhanced services not dependent on an MVPDare available. Especially, even when a broadcaster or a contentsprovider does not provide enhanced services, a third party server notdependent on the broadcaster or the contents provider may provide theenhanced services.

Moreover, according to embodiments, information search, chatting, newssearch, digital media download, product catalog download, and productpurchase, which are related to main audio and video contents, areavailable without inconvenient text input using a remote controller.

Additionally, according to embodiments, enhanced services exceeding thebandwidth limitation of a wireless channel become available.

Moreover, the above methods of the present invention can also beembodied as computer readable codes on a computer readable recordingmedium. Examples of the computer readable recording medium includeread-only memory (ROM), random-access memory (RAM), CD-ROMs, magnetictapes, floppy disks, optical data storage devices, and carrier waves(such as data transmission through the Internet).

The configurations and methods of the above-described embodiments areapplied to the video display device without any restriction, and all orsome of the embodiments may be selectively combined to have variousmodifications.

Although embodiments have been described with reference to a number ofillustrative embodiments thereof, it should be understood that numerousother modifications and embodiments can be devised by those skilled inthe art that will fall within the spirit and scope of the principles ofthis disclosure. More particularly, various variations and modificationsare possible in the component parts and/or arrangements of the subjectcombination arrangement within the scope of the disclosure, the drawingsand the appended claims. In addition to variations and modifications inthe component parts and/or arrangements, alternative uses will also beapparent to those skilled in the art.

What is claimed is:
 1. An operating method of a video display device,the method comprising: receiving a broadcasting signal through a tuner;obtaining an uncompressed audio or video (AV) content, through areceiving unit, other than the broadcasting signal; outputting theuncompressed AV content through a display unit; extractingcharacteristic information based on a genre of the uncompressed AVcontent; receiving a response to a query which includes thecharacteristic information; and outputting an enhanced service,including a service for recommending, based on the response.
 2. Theoperating method of claim 1, further comprising: outputting a userinterface on the characteristic information extraction; and receiving afirst input for setting the characteristic information extraction,wherein the user interface includes at least one of a first mode forselecting an extracting type of the characteristic information and asecond mode for selecting an extracting period of the characteristicinformation.
 3. The operating method of claim 2, wherein the first modeincludes an active mode and an inactive mode, and wherein the activemode includes at least one of an auto setting mode and a manual settingmode.
 4. The operating method of claim 3, wherein the receiving an inputfor setting the user interface includes receiving a second input forselecting a type of the characteristic information including an audiofingerprint and a video fingerprint, when the first mode sets the manualmode in the active mode.
 5. The operating method of claim 3, wherein thereceiving an input for setting the user interface includes receiving athird input on the second mode, when the first mode sets the manual modein the active mode.
 6. The operating method of claim 5, furthercomprising: setting a transmitting period of the characteristicinformation based on the third input, wherein the setting thetransmitting period includes setting the transmitting period based on adefault value or a network status, when the second mode sets an autoperiod setting mode based on the third input.
 7. The operating method ofclaim 3, wherein the auto setting mode includes at least one of an audiopriority mode that represents a mode for extracting audio characteristicinformation by priority, and a video priority mode that represents amode for extracting video characteristic information by priority.
 8. Theoperating method of claim 7, further comprising: transmitting the queryincluding the video characteristic information, when receiving theresponse corresponding to the query including the audio characteristicinformation fails, in the audio priority mode.
 9. The operating methodof claim 7, further comprising: transmitting the query including theaudio characteristic information, when receiving the responsecorresponding to the query including the video characteristicinformation fails, in the video priority mode.
 10. A video displaydevice, comprising: a tuner configured to receive a broadcasting signal;a receiving unit configured to obtain an uncompressed audio or video(AV) content, other than the broadcasting signal; a display unitconfigured to output the uncompressed AV content; and an enhancedservice management unit configured to: extract a characteristicinformation based on a genre of the uncompressed AV content, receive aresponse to a query which includes the characteristic information, andoutput an enhanced service, including a service for recommending, basedon the response.
 11. The video display device of claim 10, wherein theenhanced service management unit outputs a user interface on thecharacteristic information extraction and receives a first input forsetting the characteristic information extraction, and wherein the userinterface includes at least one of a first mode for selecting anextracting type of the characteristic information and a second mode forselecting an extracting period of the characteristic information. 12.The video display device of claim 11, wherein the first mode includes anactive mode and an inactive mode, and wherein the active mode includesat least one of an auto setting mode and a manual setting mode.
 13. Thevideo display device of claim 12, wherein the enhanced servicemanagement unit receives a second input for selecting a type of thecharacteristic information including an audio fingerprint and a videofingerprint, when the first mode sets the manual mode in the activemode.
 14. The video display device of claim 12, wherein the enhancedservice management unit receives a third input on the second mode, whenthe first mode sets the manual mode in the active mode.
 15. The videodisplay device of claim 14, wherein the enhanced service management unitsets a transmitting period of the characteristic information based onthe third input.
 16. The video display device of claim 15, wherein theenhanced service management unit sets the transmitting period based on adefault value or a network status, when the second mode sets an autoperiod setting mode based on the third input.
 17. The video displaydevice of claim 12, wherein the auto setting mode includes at least oneof an audio priority mode that represents a mode for extracting audiocharacteristic information by priority, and a video priority mode thatrepresents a mode for extracting video characteristic information bypriority.
 18. The video display device of claim 17, wherein the enhancedservice management unit transmits the query including the videocharacteristic information, when the enhanced service management unitfails to receive the response corresponding to the query including theaudio characteristic information, in the audio priority mode.
 19. Thevideo display device of claim 17, wherein the enhanced servicemanagement unit transmits the query including the audio characteristicinformation, when the enhanced service management unit fails to receivethe response corresponding to the query including the videocharacteristic information, in the video priority mode.